configure warnings in CI for libio
Hi, We have the following in CI build logs: checking libio.h usability... no | checking libio.h presence... yes || configure: WARNING: libio.h: present but cannot be compiled || configure: WARNING: libio.h: check for missing prerequisite headers? || configure: WARNING: libio.h: see the Autoconf documentation || configure: WARNING: libio.h: section "Present But Cannot Be Compiled" || configure: WARNING: libio.h: proceeding with the compiler's result || configure: WARNING: ## --- ## || configure: WARNING: ## Report this to bug-lilyp...@gnu.org ## || configure: WARNING: ## --- ## ||checking for libio.h... no Is it something we should be worried about? Best, Jean |
Re: Cairo plans
Am Dienstag, dem 31.08.2021 um 17:24 +0200 schrieb Jean Abou Samra: > To me, the top priority would be getting the fresh > Cairo backend to be tested as widely as possible, as > soon as possible. Unfortunately, I currently have some > doubt that releases with Cairo being opt-in will best > achieve this effect. The reason is that most users > are editing in Frescobaldi, and Frescobaldi doesn't > -- to my knowledge -- offer a way of permanently editing > the set of command line options passed to LilyPond > (there is "Engrave (custom)" but you have to click > OK in a new window every time, which is annoying when > working on a score). It would be best to get some users > to use Cairo for their daily engraving work, as opposed > to letting them test a few scores when the release is > out (e.g., it took me a couple months of building with Guile 2 > and developing with it to notice the \ottava quirk in > https://gitlab.com/lilypond/lilypond/-/issues/6141). > For that I think Cairo by default would be the most > effective way. Hence my earlier suggestion of making > Cairo the default in the next development release. > These are called unstable after all. I fully agree, except for the combination of "as widely as possible" and "as soon as possible". To reiterate, I propose to include it opt-in for the next stable release, and then switch early in the development cycle leading to a potential 3.0. On the other hand, I think we should definitely avoid going to Cairo by default now and then switch back before doing a stable release due to whatever reasons. In my humble opinion, that's a very good recipe for breakage, and predicting that no such reasons will occur is pure gambling. Jonas signature.asc Description: This is a digitally signed message part
Re: Cairo plans
Am Dienstag, dem 31.08.2021 um 11:34 +0200 schrieb Han-Wen Nienhuys: > > Giving timing for a single HTML file is a bit dubious because it > > requires processing all .tely files for cross-references. For the > > influence of Cairo, you really want to compare the time it takes to run > > lilypond-book to get a single .texi file. However, I'd like to remark > > that generating zillions of tiny snippets is not really the kind of > > things users tend to do... > > It is the relevant metric here, because the MR enables lilypond-book > processing. It is also relevant for development because our > regtests/CI pipelinek are based on lilypond-book. No, it's not: The timings you gave and the descriptions suggest that you get that and that speedup for producing the HTML version of the Notation Reference. This is not true because it already compiles all .tely files, so you cannot extrapolate from one manual to the entirety. The relevant metric for development is the time a complete 'make doc' takes from start to end. > > > The current SVG backend is glacially slow > > > > IIRC the reason is the widespread use of regexes for matching glyph > > nodes. I think this could be done better, and comparing speed isn't > > really fair until then. > > I think it is a fair comparison, given that it's comparing a > documented and shipping feature against an experimental feature. It > would be unfair to dismiss the experimental feature due to bugs, but > that's not what's happening here. > > The SVG backend works roughly like the PS backend (run Scheme code > that translates stencils to strings, which are dumped one by one to an > output file), so I am willing to bet serious money that you can't get > to perform faster than the PS backend, and by extension, the Cairo > backend. That's not what I was saying; the fact that you could make the SVG backend up to 5x faster to match the PS backend (not exceed the Cairo backend) makes it an unfair comparison right now. > > > Here are my questions: > > > > > > * when could/would we drop the SVG backend? > > > > > > * when could/would we switch the default PS/PDF/PNG backend to cairo? > > > > From my current understanding of missing features, the amount of > > testing the backend can have received (or rather cannot, due to > > novelty), and the nature of bugs that are found (both in Cairo itself > > and the integration in LilyPond), I don't think the backend is > > currently in a state to be used by default. I would highly prefer to > > not mix switching the default backend with switching to Guile 2.2 that > > will already be disruptive enough (yes, it's going slower than I had > > hoped...). > > I see your point, but consider the following: the PS backend > represents 3500 fiddlesome Postscript and Scheme code, which is only > exercised by us. Yes, but it's battle-tested for how long? > With the cairo solution, the complexity of rendering and font handling > is moved to Cairo, which is much more widely used and better tested. Disagreed strongly. If the addition of the Cairo backend has shown one thing, then it's the fact we're running into serious problems for using the less tested features of Cairo. It's probably working great for GUI stuff, but the file output features seem to be severely lacking (memory issues in the SVG and PNG backends, critical PDF features like links only supported since recently, forward references not working until a fix in master a few weeks ago, ...) > We are only left with cairo.cc which is much more straightforward. It > should be expected that the cairo solution takes comparatively little > effort to stabilize. > > > > * when could/would we drop the PS backend altogether? > > > > I would say that this step requires going to LilyPond 3.0, along with > > removing all the features and commands that cannot be implemented in > > the Cairo backend, or that we don't want to. > > We can discuss this in detail later, but I think a major version bump > is not warranted, as we're leaving the music input intact. For > context, the 1.8 -> 2.0 transition happened when changed > > [(c4 c4)] > > to > > c4[( c4]) > > which invalidated most .ly files, requiring manual vetting of the conversion. I stay by the fact that changing the default backend is a fundamental change that affects every single user of LilyPond. Going to 3.0 will make this clearer than any release notes could ever be. > > > I would propose the following: For the next stable release (whenever > > that will be), it would be good to have the Cairo backend mature enough > > that it can be shipped as a "preview" in the official builds (that will > > my feeling is that this state is actually pretty close now already. Maybe it is, maybe it's not - impossible to tell for a backend that is only a few weeks old. Why gamble on this? signature.asc Description: This is a digitally signed message part
Re: Cairo plans
Le 31/08/2021 à 14:47, Werner LEMBERG a écrit :
From my current understanding of missing features, the amount of
testing the backend can have received (or rather cannot, due to
novelty), and the nature of bugs that are found (both in Cairo
itself and the integration in LilyPond), I don't think the backend
is currently in a state to be used by default. I would highly
prefer to not mix switching the default backend with switching to
Guile 2.2 that will already be disruptive enough (yes, it's going
slower than I had hoped...).
I see your point, but consider the following: the PS backend
represents 3500 fiddlesome Postscript and Scheme code, which is only
exercised by us.
With the cairo solution, the complexity of rendering and font
handling is moved to Cairo, which is much more widely used and
better tested. We are only left with cairo.cc which is much more
straightforward. It should be expected that the cairo solution
takes comparatively little effort to stabilize.
In the long term this is certainly the way to go. However, applying
Jonas's conservative versioning approach makes sense to me. We aren't
in a hurry, are we?
* when could/would we drop the PS backend altogether?
I would say that this step requires going to LilyPond 3.0, along with
removing all the features and commands that cannot be implemented in
the Cairo backend, or that we don't want to.
We can discuss this in detail later, but I think a major version
bump is not warranted, as we're leaving the music input intact.
Jean's argument is a good point: Similarly to libtool, it deserves a
major version bump if you remove some functionality altogether that
was available for many years before.
My gut feeling says that a combination Guile 2.x with the Cairo
backend deserves 3.0.
I purposefully won't enter the debate about a potential
major version bump.
I would propose the following: For the next stable release
(whenever that will be), it would be good to have the Cairo backend
mature enough that it can be shipped as a "preview" in the official
builds
my feeling is that this state is actually pretty close now already.
So let's work on a new stable release, then?
To me, the top priority would be getting the fresh
Cairo backend to be tested as widely as possible, as
soon as possible. Unfortunately, I currently have some
doubt that releases with Cairo being opt-in will best
achieve this effect. The reason is that most users
are editing in Frescobaldi, and Frescobaldi doesn't
-- to my knowledge -- offer a way of permanently editing
the set of command line options passed to LilyPond
(there is "Engrave (custom)" but you have to click
OK in a new window every time, which is annoying when
working on a score). It would be best to get some users
to use Cairo for their daily engraving work, as opposed
to letting them test a few scores when the release is
out (e.g., it took me a couple months of building with Guile 2
and developing with it to notice the \ottava quirk in
https://gitlab.com/lilypond/lilypond/-/issues/6141).
For that I think Cairo by default would be the most
effective way. Hence my earlier suggestion of making
Cairo the default in the next development release.
These are called unstable after all.
An alternative would be to release separate binaries
with opt-out Cairo. They wouldn't even need to be
official. But this has the problems linked to having
several versions with the same number ("I cannot reproduce
this even with the same version as you ...").
On the lists, we see a number of power users posting
examples with \version "2.23.3". Isn't that an ideal
testing pool for Cairo?
Best,
Jean
Re: Cairo plans
> I'll look into PDF processing software. AFAICT, the subsetting > retains the character mapping of the original font, so I think it > should be possible to rewrite it to embed the Emmentaler font once, > point all font references to the full font, and elide all the > subsetted versions. Thanks. > I see no reason to keep the SVG backend alive given that Cairo > achieves the same functionality faster and with less code. I agree. >> From my current understanding of missing features, the amount of >> testing the backend can have received (or rather cannot, due to >> novelty), and the nature of bugs that are found (both in Cairo >> itself and the integration in LilyPond), I don't think the backend >> is currently in a state to be used by default. I would highly >> prefer to not mix switching the default backend with switching to >> Guile 2.2 that will already be disruptive enough (yes, it's going >> slower than I had hoped...). > > I see your point, but consider the following: the PS backend > represents 3500 fiddlesome Postscript and Scheme code, which is only > exercised by us. > > With the cairo solution, the complexity of rendering and font > handling is moved to Cairo, which is much more widely used and > better tested. We are only left with cairo.cc which is much more > straightforward. It should be expected that the cairo solution > takes comparatively little effort to stabilize. In the long term this is certainly the way to go. However, applying Jonas's conservative versioning approach makes sense to me. We aren't in a hurry, are we? >> > * when could/would we drop the PS backend altogether? >> >> I would say that this step requires going to LilyPond 3.0, along with >> removing all the features and commands that cannot be implemented in >> the Cairo backend, or that we don't want to. > > We can discuss this in detail later, but I think a major version > bump is not warranted, as we're leaving the music input intact. Jean's argument is a good point: Similarly to libtool, it deserves a major version bump if you remove some functionality altogether that was available for many years before. My gut feeling says that a combination Guile 2.x with the Cairo backend deserves 3.0. >> I would propose the following: For the next stable release >> (whenever that will be), it would be good to have the Cairo backend >> mature enough that it can be shipped as a "preview" in the official >> builds > > my feeling is that this state is actually pretty close now already. So let's work on a new stable release, then? Werner
Re: Cairo plans
Le 31/08/2021 à 12:37, Kevin Barry a écrit : I would also have been inclined to say this suggests a major version bump. But if what you say is true - that input is completely unchanged - then it may not be necessary. Will the loss of the ps-command affect users? Basically, the \postscript markup command will no longer work (though \epsfile could be made to work). This is not deemed so bad because it is a security vulnerability (one can inject arbitrary PS code). Best, Jean
Re: Cairo plans
These changes/improvements are exciting. Thank you and Knut for the hard work! > > I would say that this step requires going to LilyPond 3.0, along with > > removing all the features and commands that cannot be implemented in > > the Cairo backend, or that we don't want to. > > We can discuss this in detail later, but I think a major version bump > is not warranted, as we're leaving the music input intact. I would also have been inclined to say this suggests a major version bump. But if what you say is true - that input is completely unchanged - then it may not be necessary. Will the loss of the ps-command affect users? Kevin
Re: Cairo plans
On Mon, Aug 30, 2021 at 6:31 PM Jonas Hahnfeld wrote: > > Am Montag, dem 30.08.2021 um 10:16 +0200 schrieb Han-Wen Nienhuys: > > With MR 897, I have been able to run the doc build through > > Cairo. Results are very encouraging: the build is faster, while the > > resulting PDF file is smaller. > > As Werner remarked, only due to not using extractpdfmark. I'll look into PDF processing software. AFAICT, the subsetting retains the character mapping of the original font, so I think it should be possible to rewrite it to embed the Emmentaler font once, point all font references to the full font, and elide all the subsetted versions. > Giving timing for a single HTML file is a bit dubious because it > requires processing all .tely files for cross-references. For the > influence of Cairo, you really want to compare the time it takes to run > lilypond-book to get a single .texi file. However, I'd like to remark > that generating zillions of tiny snippets is not really the kind of > things users tend to do... It is the relevant metric here, because the MR enables lilypond-book processing. It is also relevant for development because our regtests/CI pipelinek are based on lilypond-book. We discussed the performance impact of Cairo earlier, but here are timings for the MSDM score, CPU pinned at 2Ghz. -dbackend=ps --ps real 0m33.043s --pdf real 0m34.241s -dbackend=cairo --ps real 0m30.772s --pdf real 0m31.245s -dbackend=svg real 2m47.836s > > Open question is how to position Cairo output and what defaults we > > should provide. > > > > * SVG. > > > > The current SVG backend is glacially slow > > IIRC the reason is the widespread use of regexes for matching glyph > nodes. I think this could be done better, and comparing speed isn't > really fair until then. I think it is a fair comparison, given that it's comparing a documented and shipping feature against an experimental feature. It would be unfair to dismiss the experimental feature due to bugs, but that's not what's happening here. The SVG backend works roughly like the PS backend (run Scheme code that translates stencils to strings, which are dumped one by one to an output file), so I am willing to bet serious money that you can't get to perform faster than the PS backend, and by extension, the Cairo backend. From a quick look, the regexp stuff could be mitigated by doing proper XML parsing, but then you'd have to add libxml bindings for GUILE to the build. However, my main point is that the SVG output is extremely slow, and in a poor state of maintenance. The missing feature (relative to Cairo) is SVG WOFF support, but: [hanwen@localhost lilypond]$ git describe HEAD release/2.20.0-1-18-g2a3a1ed05c [hanwen@localhost lilypond]$ time lilypond -dbackend=svg -dsvg-woff input/regression/les-nereides.ly GNU LilyPond 2.20.0 ... Layout output to `les-nereides.svg'.../home/hanwen/vc/lilypond/out/share/lilypond/current/scm/output-svg.scm:455:16: While evaluating arguments to ly:paper-get-font in expression (ly:paper-get-font paper (quasiquote #)): /home/hanwen/vc/lilypond/out/share/lilypond/current/scm/output-svg.scm:455:16: Unbound variable: paper After looking through our bugs, it turns out this has been broken for at least 5 years now, (see https://gitlab.com/lilypond/lilypond/-/issues/4648) I see no reason to keep the SVG backend alive given that Cairo achieves the same functionality faster and with less code. > > Here are my questions: > > > > * when could/would we drop the SVG backend? > > > > * when could/would we switch the default PS/PDF/PNG backend to cairo? > > From my current understanding of missing features, the amount of > testing the backend can have received (or rather cannot, due to > novelty), and the nature of bugs that are found (both in Cairo itself > and the integration in LilyPond), I don't think the backend is > currently in a state to be used by default. I would highly prefer to > not mix switching the default backend with switching to Guile 2.2 that > will already be disruptive enough (yes, it's going slower than I had > hoped...). I see your point, but consider the following: the PS backend represents 3500 fiddlesome Postscript and Scheme code, which is only exercised by us. With the cairo solution, the complexity of rendering and font handling is moved to Cairo, which is much more widely used and better tested. We are only left with cairo.cc which is much more straightforward. It should be expected that the cairo solution takes comparatively little effort to stabilize. > > * when could/would we drop the PS backend altogether? > > I would say that this step requires going to LilyPond 3.0, along with > removing all the features and commands that cannot be implemented in > the Cairo backend, or that we don't want to. We can discuss this in detail later, but I think a major version bump is not warranted, as we're leaving the music input intact. For context, the 1.8 -> 2.0 transition happened when changed [(c4 c4)] to
Re: Cairo plans
Jonas Hahnfeld writes: > Am Montag, dem 30.08.2021 um 18:47 +0200 schrieb David Kastrup: >> Jonas Hahnfeld via Discussions on LilyPond development >> writes: >> >> > Giving timing for a single HTML file is a bit dubious because it >> > requires processing all .tely files for cross-references. For the >> > influence of Cairo, you really want to compare the time it takes to >> > run lilypond-book to get a single .texi file. However, I'd like to >> > remark that generating zillions of tiny snippets is not really the >> > kind of things users tend to do... >> >> It's likely the workload for Wikimedia (though likely not via PDF) where >> performance is pretty relevant. > > Maybe, but (please correct me if I'm wrong) Wikimedia is not compiling > zillions of small scores via a single process as the doc build is doing > for the NR. And once you start a new process for every snippet and need > to wait 0.6 seconds for LilyPond to boot up, the speed advantage of > Cairo compared to the existing backends becomes negligible, I suppose. I'd imagine that starting up GhostScript with font loading and stuff for every image or not would still make a difference. -- David Kastrup
Re: Cairo plans
Am Montag, dem 30.08.2021 um 18:47 +0200 schrieb David Kastrup: > Jonas Hahnfeld via Discussions on LilyPond development > writes: > > > Giving timing for a single HTML file is a bit dubious because it > > requires processing all .tely files for cross-references. For the > > influence of Cairo, you really want to compare the time it takes to > > run lilypond-book to get a single .texi file. However, I'd like to > > remark that generating zillions of tiny snippets is not really the > > kind of things users tend to do... > > It's likely the workload for Wikimedia (though likely not via PDF) where > performance is pretty relevant. Maybe, but (please correct me if I'm wrong) Wikimedia is not compiling zillions of small scores via a single process as the doc build is doing for the NR. And once you start a new process for every snippet and need to wait 0.6 seconds for LilyPond to boot up, the speed advantage of Cairo compared to the existing backends becomes negligible, I suppose. signature.asc Description: This is a digitally signed message part
Re: Cairo plans
On Tue, Aug 31, 2021 at 7:13 AM Jan Nieuwenhuizen wrote: > > Han-Wen Nienhuys writes: > > Hi, > > > Also: apologies to reviewers for the large Merge-Request. > > Unfortunately, the backend code was quite convoluted, and I didn't see > > a way except to just slash my way through it. refactoring along the > > way. > > Sounds good, can I have a look at the patch set, do you have a git > branch somewhere? I checked some projects for usage of \ps-command and > it doesn't seem to bo popular. Losing that would have been my main > concern, I think. it's here, https://gitlab.com/lilypond/lilypond/-/tree/dev/hanwen/cairo-lp-book You can download with git fetch https://gitlab.com/lilypond/lilypond.git dev/hanwen/cairo-lp-book:foo -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
