Re: [RFC] Transition to Guile 3.0
Am 08.04.2024 um 23:40 schrieb Jonas Hahnfeld: Thanks for testing! I assume this is also enabling Guile optimizations during LilyPond runtime? It would be interesting to see if there's a gain from just compiling the bytecode with optimizations. That would be a one-time cost that may be worth paying, especially if we had proper standalone bytecode compilation that parallelizes... Btw, I get a mass of warning messages during bytecode compilation about possibly unbound variables. IIUC, the reason is that we do not import our "own" modules (the ones in (lily)) from the files that need them, rather relying on loading all modules during startup one after another. Is this correct? That would explain that Guile fails to sort out the dependencies between the modules. Michael
Re: [RFC] Transition to Guile 3.0
Am 08.04.2024 um 23:40 schrieb Jonas Hahnfeld: [snip] Thanks for testing! I assume this is also enabling Guile optimizations during LilyPond runtime? It would be interesting to see if there's a gain from just compiling the bytecode with optimizations. That would be a one-time cost that may be worth paying, especially if we had proper standalone bytecode compilation that parallelizes... IIUC, this will only enable Guile optimizations during runtime when calling LilyPond with -dcompile-scheme-code=#t. My measurements were done without this option, so this is already the case you mentioned. Out of curiosity, I tested setting the optimization level to the maximum value (9). Bytecode compilation speed is exactly the same as with the default (2), runtime speed is not better than with level 2, however. Btw, I'm curious if -dcompile-scheme-code will work on Windows with your recent fixes for Guile on mingw64. Currently, it even segfaults for me. Michael
Re: [RFC] Transition to Guile 3.0
On Tue, 2024-04-02 at 16:40 +0200, Michael Käppler wrote: > Am 01.04.2024 um 22:03 schrieb Jonas Hahnfeld via Discussions on LilyPond > development: > > As pointed out by Han-Wen in November, this is actually fairly little > > code that gets dropped; we need to keep some related to optimizations > > for compatibility with 3.0.0 up to 3.0.2; see also below. > > Just to share some numbers: It is still the case with current Guile master, > that bytecode compilation is nearly 10x slower with the default optimization > level than > with optimizations switched off. > > make bytecode with default optimization level: > real 2m0,875s > user 1m59,989s > sys 0m0,796s > > make bytecode without optimizations: > real 0m16,535s > user 0m14,209s > sys 0m0,810s > > What concerns LilyPond processing speed, I did a comparison with the > medium-sized score > I also used for the JIT vs. non-JIT timings on Windows and got for 10 > subsequent runs: > > Without optimizations: > real 9m1,511s > user 8m52,375s > sys 0m11,423s > > With default optimization level: > real 8m34,762s > user 8m23,027s > sys 0m14,562s > > That would be -5% in compilation time. Does not pay off, I think. Thanks for testing! I assume this is also enabling Guile optimizations during LilyPond runtime? It would be interesting to see if there's a gain from just compiling the bytecode with optimizations. That would be a one-time cost that may be worth paying, especially if we had proper standalone bytecode compilation that parallelizes... Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Transition to Guile 3.0
Am 01.04.2024 um 22:03 schrieb Jonas Hahnfeld via Discussions on LilyPond development: This is now up for review in the following merge request: https://gitlab.com/lilypond/lilypond/-/merge_requests/2293 As pointed out by Han-Wen in November, this is actually fairly little code that gets dropped; we need to keep some related to optimizations for compatibility with 3.0.0 up to 3.0.2; see also below. Just to share some numbers: It is still the case with current Guile master, that bytecode compilation is nearly 10x slower with the default optimization level than with optimizations switched off. make bytecode with default optimization level: real 2m0,875s user 1m59,989s sys 0m0,796s make bytecode without optimizations: real 0m16,535s user 0m14,209s sys 0m0,810s What concerns LilyPond processing speed, I did a comparison with the medium-sized score I also used for the JIT vs. non-JIT timings on Windows and got for 10 subsequent runs: Without optimizations: real 9m1,511s user 8m52,375s sys 0m11,423s With default optimization level: real 8m34,762s user 8m23,027s sys 0m14,562s That would be -5% in compilation time. Does not pay off, I think. Michael
Re: [RFC] Transition to Guile 3.0
On Sun, 2023-11-05 at 22:36 +0100, Jonas Hahnfeld wrote: > Step 4: Remove compatibility code for Guile 2.2 > This can happen after we made one or two releases with only Guile 3.0. This is now up for review in the following merge request: https://gitlab.com/lilypond/lilypond/-/merge_requests/2293 As pointed out by Han-Wen in November, this is actually fairly little code that gets dropped; we need to keep some related to optimizations for compatibility with 3.0.0 up to 3.0.2; see also below. > Step 5? Move to Ubuntu 22.04 for testing and bump requirement to Guile > 3.0.7? Then we could remove some more compatibility code, but let's > discuss this at some later point. I still want to go there, pending concrete assessment and implementation. Cheers Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Transition to Guile 3.0
On Mon, 2023-12-04 at 21:55 +0100, Jonas Hahnfeld wrote: > On Sun, 2023-11-05 at 22:36 +0100, Jonas Hahnfeld wrote: > > > then remove automatic detection of Guile 2.2 from configure. > > I did not yet upload a merge request to make Guile 3.0 the default for > all builds: After more thought, I believe it's better to do this > together with the removal of automatic configure support for Guile 2.2, > to avoid the situation where we assume that Guile 3.0 is the default, > but building from source will gracefully configure with Guile 2.2, for > example because the system is missing the development package for Guile > 3.0... This next step (requiring Guile 3.0 and removing automatic detection of Guile 2.2 from configure) is now available as https://gitlab.com/lilypond/lilypond/-/merge_requests/2228 Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Transition to Guile 3.0
>> then remove automatic detection of Guile 2.2 from configure. > > I did not yet upload a merge request to make Guile 3.0 the default > for all builds: After more thought, I believe it's better to do this > together with the removal of automatic configure support for Guile > 2.2, to avoid the situation where we assume that Guile 3.0 is the > default, but building from source will gracefully configure with > Guile 2.2, for example because the system is missing the development > package for Guile 3.0... +1 Werner
Re: [RFC] Transition to Guile 3.0
On Sun, 2023-11-05 at 22:36 +0100, Jonas Hahnfeld wrote: > Step 3: Switch to Guile 3.0 > Afterwards, we can merge > https://gitlab.com/lilypond/lilypond/-/merge_requests/2163 I put the merge request back on Patch::review, along with https://gitlab.com/lilypond/lilypond/-/merge_requests/2170 for the documentation build. Then we could have a release next weekend or the one after. If next weekend, the first release in January might be a bit bigger, we'll have to see... > then remove automatic detection of Guile 2.2 from configure. I did not yet upload a merge request to make Guile 3.0 the default for all builds: After more thought, I believe it's better to do this together with the removal of automatic configure support for Guile 2.2, to avoid the situation where we assume that Guile 3.0 is the default, but building from source will gracefully configure with Guile 2.2, for example because the system is missing the development package for Guile 3.0... Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Transition to Guile 3.0
On Sat, 2023-11-11 at 18:37 +0100, Han-Wen Nienhuys wrote: > On Sun, Nov 5, 2023 at 10:36 PM Jonas Hahnfeld wrote: > > Hi all, > > > > I hear LilyPond hasn't changed its Guile version since some time (more > > than 18 months). So before we get too comfortable with the current > > situation, let me propose to move to Guile 3.0. Below is a plan for > > that switch, with a transition period to test the official binaries. > > Last time, when going from Guile 1.8 to version 2.2, the switch had to > > coincide with moving away from GUB. Between Guile 2.2 and 3.0, we could > > in principle support both versions for a longer period. However, I > > personally think that a full transition and dropping support for Guile > > 2.2 is the more reasonable approach: It will reduce testing > > configurations (both for development and user reports) and hopefully > > enable some future cleanups in the code. > > Could you say a bit more about the benefits/disadvantages for the user? > I had the impression that Guile 3 had different (worse?) performance > characteristics relative to 2.2, but it's been a while. As far as I can tell from my limited tests so far, LilyPond built against Guile 2.2 or 3.0 has similar speeds. At least if you have compiled bytecode (that generates faster with Guile 3.0, but that's not really an advantage for users). The build with Guile 3.0 seems to have a slightly lower startup time, which is good for small scores, but we will be able to test in full tomorrow when I build the other binaries. > IIRC, one of the arguments to drop 1.8 is that Guile pre-2.x did not > support installing multiple versions alongside each other, which forced > distributions to choose whether to ship LilyPond or a recent version of > Guile. With 2.2 and later, that dilemma disappeared. Not quite: While it's true that Guile 1.8 was not prepared to be co- installed with later versions, it worked in practice because Guile 2.0 had the necessary modifications. What happened instead is that Linux distributions wanted to get rid of Guile 1.8, and some did before LilyPond was able to fully work with Guile 2.2. The same is currently happening again, for example we had to do quite some convincing to keep Guile 2.2 in Debian 12 Bookworm released this summer. So IMHO we really want to support Guile 3.0; Jean already did the hard work here last year, up to the point that some already build LilyPond 2.24.x with Guile 3.0 right now. > I quickly grepped over the source (grepping for SCM_MAJOR_VERSION), > but the bifurcations look very modest. In principle yes, we could continue to support Guile 2.2 and it wouldn't cause too many issues at first. However, we know from experience that supporting two versions of Guile has a certain cost, and I think we are better off just supporting Guile 3.0 going forward. Jonas signature.asc Description: This is a digitally signed message part
Re: [RFC] Transition to Guile 3.0
On Sun, Nov 5, 2023 at 10:36 PM Jonas Hahnfeld via Discussions on LilyPond development wrote: > Hi all, > > I hear LilyPond hasn't changed its Guile version since some time (more > than 18 months). So before we get too comfortable with the current > situation, let me propose to move to Guile 3.0. Below is a plan for > that switch, with a transition period to test the official binaries. > Last time, when going from Guile 1.8 to version 2.2, the switch had to > coincide with moving away from GUB. Between Guile 2.2 and 3.0, we could > in principle support both versions for a longer period. However, I > personally think that a full transition and dropping support for Guile > 2.2 is the more reasonable approach: It will reduce testing > configurations (both for development and user reports) and hopefully > enable some future cleanups in the code. > Could you say a bit more about the benefits/disadvantages for the user? I had the impression that Guile 3 had different (worse?) performance characteristics relative to 2.2, but it's been a while. IIRC, one of the arguments to drop 1.8 is that Guile pre-2.x did not support installing multiple versions alongside each other, which forced distributions to choose whether to ship LilyPond or a recent version of Guile. With 2.2 and later, that dilemma disappeared. I quickly grepped over the source (grepping for SCM_MAJOR_VERSION), but the bifurcations look very modest. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: [RFC] Transition to Guile 3.0
>> Please let me know of your comments! > > I'm very happy that you got Guile 3.0 working on Windows. Kudos for > that (and I guess we need to send big thanks to Mike Gran too). +1 > What I don't really understand is why you want to add compatibility > with Guile 3.0.x for small x. [...] I think what Jonas does is the right way – support for Guile 3 in general to accommodate with as many distributions as possible. Werner
Re: [RFC] Transition to Guile 3.0
Le lundi 06 novembre 2023 à 01:11 +0100, David Kastrup a écrit : > I have not checked recently, but last time I looked, Guile's versioning > system more or less worked so that the development branch (and the > corresponding versions) was Andy Wingo's playground. It was not > concerned with APIs in any manner. Any API development and > corresponding changes consequently had to be done in the stable branch > and eventually forwardported into the development branch. Currently, as far as I am aware, there is not even a development branch separate from the stable branch. The branch "main" receives all development and 3.0.x releases are simply cut from it. That's even worse for stability than the setup you describe. signature.asc Description: This is a digitally signed message part
Re: [RFC] Transition to Guile 3.0
Jean Abou Samra writes: > What I don't really understand is why you want to add compatibility > with Guile 3.0.x for small x. Upstream completely breaks the normal > expectation from what you would find in a point release, by putting > features and even severely backwards incompatible changes (like > cross-module inlining in 3.0.8 *cough*), so I think this is definitely > going to be more work to keep supporting (in particular: testing) than > with dependencies that don't have this peculiarity. I'd rather define > a minimum version in the 3.0.x series that we want to support. I have not checked recently, but last time I looked, Guile's versioning system more or less worked so that the development branch (and the corresponding versions) was Andy Wingo's playground. It was not concerned with APIs in any manner. Any API development and corresponding changes consequently had to be done in the stable branch and eventually forwardported into the development branch. If that is still how things are done, the kind of feature/API stability we can depend on from distributions based on how much they can depend on being able to rely on version numbers for stability choices is limited. We use Guile as infrastructure, not as a Scheme hacking playground. But while Guile is historically sold as infrastructure, its community setup does not accommodate it. And for resolving that mess, we as the likely most competent party with an interest in that mess will be tasked with the consequences, for better or worse. -- David Kastrup
Re: [RFC] Transition to Guile 3.0
On Sun, 2023-11-05 at 22:48 +0100, Jean Abou Samra wrote: > What I don't really understand is why you want to add compatibility > with Guile 3.0.x for small x. Upstream completely breaks the normal > expectation from what you would find in a point release, by putting > features and even severely backwards incompatible changes (like > cross-module inlining in 3.0.8 *cough*), so I think this is definitely > going to be more work to keep supporting (in particular: testing) than > with dependencies that don't have this peculiarity. I'd rather define > a minimum version in the 3.0.x series that we want to support. Sure, but that is currently defined by Ubuntu 20.04 having Guile 3.0.1. We could of course move "Step 5" earlier in the process, but then I argue we still need to pay attention to what versions of Guile are in the wild, the next one probably being Guile 3.0.5 in Debian Bullseye... signature.asc Description: This is a digitally signed message part
Re: [RFC] Transition to Guile 3.0
Le dimanche 05 novembre 2023 à 22:36 +0100, Jonas Hahnfeld via Discussions on LilyPond development a écrit : > Please let me know of your comments! I'm very happy that you got Guile 3.0 working on Windows. Kudos for that (and I guess we need to send big thanks to Mike Gran too). What I don't really understand is why you want to add compatibility with Guile 3.0.x for small x. Upstream completely breaks the normal expectation from what you would find in a point release, by putting features and even severely backwards incompatible changes (like cross-module inlining in 3.0.8 *cough*), so I think this is definitely going to be more work to keep supporting (in particular: testing) than with dependencies that don't have this peculiarity. I'd rather define a minimum version in the 3.0.x series that we want to support. Jean signature.asc Description: This is a digitally signed message part
