Re: Blockers for Guile 2.2
Am Sonntag, dem 20.03.2022 um 15:48 +0100 schrieb Jonas Hahnfeld via Discussions on LilyPond development: > Am Donnerstag, dem 17.03.2022 um 22:38 +0100 schrieb Jonas Hahnfeld via > Discussions on LilyPond development: > > Am Donnerstag, dem 17.03.2022 um 22:10 +0100 schrieb Jean Abou Samra: > > > > Le 17 mars 2022 à 20:35, Jonas Hahnfeld > > > > > > > Now merged. And while you can continue building with Guile 1.8 (for > > > > now), I hope we can remove that code soon after the next release. So > > > > please don't do this and switch your builds to Guile 2.2 > > > > > > One more thing: I suppose you want an unstable release to happen rather > > > sooner than later now there we're on Guile 2, is there a moment planned? > > > This is a mere curiosity question. > > > > maybe this weekend > > I'm going to try this now, but no guarantees that I'll manage. If I > don't, no problem since I don't need to push until I'm sure that > everything worked out. In the end, I think everything worked fine so far from the release side and in principle, I have all binaries ready. However, after giving them a quick round of testing, I discovered two issues that I think would be good to fix: https://gitlab.com/lilypond/lilypond/-/merge_requests/1253 Especially the version printout might cause problems if people think that this is the new format and start updating some regular expression. So while I could upload the release tomorrow, I tend towards not doing it unless somebody really wants me to. I will try again next weekend. Jonas signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Am Donnerstag, dem 17.03.2022 um 22:38 +0100 schrieb Jonas Hahnfeld via Discussions on LilyPond development: > Am Donnerstag, dem 17.03.2022 um 22:10 +0100 schrieb Jean Abou Samra: > > > Le 17 mars 2022 à 20:35, Jonas Hahnfeld > > > > > Now merged. And while you can continue building with Guile 1.8 (for > > > now), I hope we can remove that code soon after the next release. So > > > please don't do this and switch your builds to Guile 2.2 > > > > One more thing: I suppose you want an unstable release to happen rather > > sooner than later now there we're on Guile 2, is there a moment planned? > > This is a mere curiosity question. > > Yes, and unless somebody objects with good reasons, I'd like to manage > the releases for some time now. To clarify, by actually using the scripts I will be able to work out and document the updated release procedure. Some actions that GUB did in the background will become explicit now, and there will surely be problems down the road that I cannot anticipate until I run into them. > maybe this weekend I'm going to try this now, but no guarantees that I'll manage. If I don't, no problem since I don't need to push until I'm sure that everything worked out. Jonas signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
> [...] I'd like to manage the releases for some time now. +1, and thanks! Werner
Re: Blockers for Guile 2.2
Le 18/03/2022 à 08:38, Jonas Hahnfeld a écrit : Am Freitag, dem 18.03.2022 um 00:55 +0100 schrieb Jean Abou Samra: One other thought in passing: should we write to maintainers of popular distros to let them know that we've switched to Guile 2? What would you write them at the current point in time? That current master has switched to only looking for Guile 2.2, which hasn't even made it into a development release? After 2.23.7 is out, do we want all distributions to package that? I don't think this makes sense. No, I was just thinking that doing the switch in their packages could be a nontrivial task and they might want to know in advance rather than discovering it on the day we release 2.24.
Re: Blockers for Guile 2.2
Am Freitag, dem 18.03.2022 um 00:55 +0100 schrieb Jean Abou Samra: > One other thought in passing: should we > write to maintainers of popular distros to let them know > that we've switched to Guile 2? What would you write them at the current point in time? That current master has switched to only looking for Guile 2.2, which hasn't even made it into a development release? After 2.23.7 is out, do we want all distributions to package that? I don't think this makes sense. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Just wanted to say this is great L
Re: Blockers for Guile 2.2
Le 17/03/2022 à 22:38, Jonas Hahnfeld a écrit : Am Donnerstag, dem 17.03.2022 um 22:10 +0100 schrieb Jean Abou Samra: Le 17 mars 2022 à 20:35, Jonas Hahnfeld One more thing: I suppose you want an unstable release to happen rather sooner than later now there we're on Guile 2, is there a moment planned? This is a mere curiosity question. Yes, and unless somebody objects with good reasons, I'd like to manage the releases for some time now. (So far, I may have often done some preparatory work, but Phil did the actual release commits, built using GUB and uploaded the results.) But no concrete date planned yet, we (Han-Wen and me) need to discuss how the source tarball gets uploaded to lilypond.org and how to update the online documentation. If that goes really fast, maybe this weekend but rather unlikely, I would say. OK, thanks. One other thought in passing: should we write to maintainers of popular distros to let them know that we've switched to Guile 2?
Re: Blockers for Guile 2.2
Am Donnerstag, dem 17.03.2022 um 22:10 +0100 schrieb Jean Abou Samra: > > Le 17 mars 2022 à 20:35, Jonas Hahnfeld > > > Now merged. And while you can continue building with Guile 1.8 (for > > now), I hope we can remove that code soon after the next release. So > > please don't do this and switch your builds to Guile 2.2 > > One more thing: I suppose you want an unstable release to happen rather > sooner than later now there we're on Guile 2, is there a moment planned? > This is a mere curiosity question. Yes, and unless somebody objects with good reasons, I'd like to manage the releases for some time now. (So far, I may have often done some preparatory work, but Phil did the actual release commits, built using GUB and uploaded the results.) But no concrete date planned yet, we (Han-Wen and me) need to discuss how the source tarball gets uploaded to lilypond.org and how to update the online documentation. If that goes really fast, maybe this weekend but rather unlikely, I would say. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
> Le 17 mars 2022 à 20:35, Jonas Hahnfeld > Now merged. And while you can continue building with Guile 1.8 (for > now), I hope we can remove that code soon after the next release. So > please don't do this and switch your builds to Guile 2.2 One more thing: I suppose you want an unstable release to happen rather sooner than later now there we're on Guile 2, is there a moment planned? This is a mere curiosity question. Welcome in a new age of LilyPond's history.
Re: Blockers for Guile 2.2
Am Mittwoch, dem 16.03.2022 um 17:30 +0100 schrieb Jean Abou Samra: > Le 19/02/2022 à 17:57, Jonas Hahnfeld a écrit : > > Hi all, > > > > I'd like to discuss what are considered blocker issues for a switch to > > Guile 2.2. > > > > [...] > > Pasting this link for future reference: > > https://gitlab.com/lilypond/lilypond/-/merge_requests/1250 Now merged. And while you can continue building with Guile 1.8 (for now), I hope we can remove that code soon after the next release. So please don't do this and switch your builds to Guile 2.2 Jonas signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Le 19/02/2022 à 17:57, Jonas Hahnfeld a écrit : Hi all, I'd like to discuss what are considered blocker issues for a switch to Guile 2.2. After the release of 2.23.6, there were reports of major problems on Windows, namely that the binaries were broken when extracted with the Windows Explorer (#6281) and that file names with special characters didn't work (#6282). I think I found solutions for both of them, either already merged (!1194 for #6281) or in review (!1219 for #6282). The second large topic was performance of the binaries with Guile 2.2, which we know will be worse without compiled bytecode. In https://lists.gnu.org/archive/html/lilypond-devel/2022-02/msg00099.html Jean writes [Guile bytecode for LilyPond's .scm files] should be added eventually before we make a full switch. I don't fully agree and think that bytecode compilation shouldn't block the switch. In my opinion, it would be fine for the next development releases to be somewhat slower if that results in Guile 2.2 being available sooner. The trade-off will be different for a stable release where we certainly want LilyPond to be as fast as possible. That was the main reason why I posted about GUILE_AUTO_COMPILE last year, to have a fallback ready in case we can't get proper compilation working (and showing that the compiled bytecode gives very similar performance to Guile 1.8, of course). What do you think? Are there are other issues that need to be taken into account? Jonas Pasting this link for future reference: https://gitlab.com/lilypond/lilypond/-/merge_requests/1250 Jean
Re: Blockers for Guile 2.2
Le 26/02/2022 à 15:10, Jonas Hahnfeld a écrit : Am Samstag, dem 26.02.2022 um 14:47 +0100 schrieb Jean Abou Samra: Le 26/02/2022 à 13:51, Han-Wen Nienhuys a écrit : The Scheme compilation felt much slower, and for C++ ccache takes away a lot of the pain of recompiles. It also appears to be single-threaded? I admit not having timed it in detail. OK, I have very good news regarding compilation speed. Tests are done with rm -rf out/share/lilypond/current/guile/ && time out/bin/lilypond (I have GUILE_AUTO_COMPILE=1 in my environment.) * master and Guile 2 real 0m58,877s user 0m58,773s sys 0m0,124s Execution time on MSDM.ly: real 0m18,870s user 0m18,727s sys 0m0,470s * Guile 2, with attached patch disabling all optimizations: He, I always thought auto-compilation didn't optimize! now don't tell me Guile also applies optimizations while just reading and supposedly interpreting code... real 0m16,791s user 0m16,700s sys 0m0,056s Execution time on MSDM.ly: real 0m18,702s user 0m18,517s sys 0m0,509s [ skipping over the part regarding Guile 3, since I don't think it's relevant here ] For one thing, Guile's optimization make about zero difference for the speed of the resulting LilyPond executable. For another, disabling optimizations in Guile 2 already results in a good speedup (1min to 20s), [...]. [...] What do you think? Yes, it looks like we should do this! On the patch, I think it would be better to apply the strategy from module/scripts/compile.scm and just get all available-optimizations from the concatenation of tree-il- default-optimization-options and cps-default-optimization-options instead of hard-coding the list. Jonas https://gitlab.com/lilypond/lilypond/-/merge_requests/1247
Re: Blockers for Guile 2.2
Am Freitag, dem 25.02.2022 um 08:18 +0100 schrieb Jonas Hahnfeld via Discussions on LilyPond development: > Am Freitag, dem 25.02.2022 um 00:08 +0100 schrieb Jean Abou Samra: > > > > > > See https://gitlab.com/hahnjo/lilypond/-/commits/guile2-bytecode Let me > > > know if this is miraculously sufficient to make people happy and I can > > > open a merge request. > > > > I was going to write a longer reply before this came, but > > this mostly obviates it. Am I understanding it correctly that > > you can copy the bytecode from Linux binaries to cross-compiled > > MingW ones even if this is not automated for now? > > That is my understanding and the claim I'm making since last year, but > I haven't tested this yet. Yes, works exactly as expected: https://gitlab.com/lilypond/lilypond/-/merge_requests/1239 signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
On Sun, Feb 27, 2022 at 12:13 PM Han-Wen Nienhuys wrote: > On Sun, Feb 27, 2022 at 10:39 AM Luca Fascione > wrote: > > is it true that if you double the source size you double the compilation > time? > > it should be, but we have rather complicated page breaking code that > is so hairy that nobody understands it fully. I'm not sure there is > NP-complete snake in the proverbial grass there. > Understood. As a use of both lilypond and LaTeX I have been idly wondering for years whether modern computers can afford to use global line/page breaking algorithms that would avoid some of the shortcomings of TeX's approach. A discussion for a different thread, of course. accessing data (eg. offsets): > * on first access, the callback gets executed. This is just evaluating > ( ). > * on second access, the value is cached in an alist, and looking it up > is extremely cheap. > This is cool. :-) I don't know enough about this program to even begin to have a gut feeling, however I guess I'm thinking it seems there would be tons of these reads, and I'm hearing you say that in an eventual sense, all data access is an alist access. I don't know how alists are actually implemented under the hood, but they feel like they would be a linear scan with symbol-typed keys on the surface. So to pull out one float you're doing what 5-10 64bit compares (checking the keys) and just as many pointer jumps, right? (I'm thinking the alist is a list of symbol/value pairs in the implementation also). This cost strongly dominates the float dereference itself, and there is this question of how much extra stuff is happening around it (I guess in my mind I'm comparing it to element access in C++ which is one pointer (this), one offset for the member (compiled into an immediate), one load of this+offset (which the hardware helps you with)). I feel for the moment I can't provide any concrete insight into any of this, because I don't know the specifics enough. > Code following a simple pattern like this, once compiled, will largely be > dominated by the > > scripting language runtime overhead > > From the outside this may seem plausible, but I doubt your intuition here: > > * the callback tables are also alists. They are cheap (they aren't > sped up if you swap them for hash tables) > Not presuming to know your program better than you, but I'd just bring up that this is saying that your lists are short (likely length 20ish on average): the observation you report is that hashing so you can do a direct access into an array is not faster than several pointer-pointer comparisons and pointer chases. The hash you'd use here be something like FNV or so, it'll break even somewhere in the 10-20 comparisons, I'd expect. > * Scheme has no marshaling: objects are either direct (scheme -> C++ > is bit twiddling), or they are indirect (a tagged pair with a C++ > pointer in the cdr) > Half of that I expected (more specifically, for various reasons, a number of which not accurate, I expected the Scheme APIs to be similar to the TCL APIs, and there as well you just get handed straight what TCL has in hand, not marshalling involved). One thing I didn't know is that the client calls to extract the machine representation of the value would be super cheap. But still, if the guile compiler translates scheme values into native ones and is able to leave them there for "long" stretches of code in some cases, and our use case instead prevents that, it seems it could eventually add up. Again I do need to learn the source better before you give these thoughts any real weight. > IMO The real problem is that we don't have good tooling to see what is > going on in the Scheme side of things: C++ has decent perf analysis, > but the Guile side of things just looks like a lot of time spent in > scm_eval(). Some of it is overhead, some if it might be a poorly > implemented Scheme function (which is often easier to fix than > reducing overhead.) > Very agreed that poorly conceived code is the first thing to address, no doubt. I'd think that the way to gain insight as to what's going on is to inspect the bytecode actually, and gain familiarity with the APIs that execute it. Is it that the bytecode is then translated to executable, or is it running on a VM? I would assume they don't provide a decompiler of any sort, do they? Thanks for a most interesting discussion L
Re: Blockers for Guile 2.2
On Sat, Feb 26, 2022 at 2:02 PM Jonas Hahnfeld wrote: > > > The same happens for C++ files, you also have to recompile. But it's > > > true that editing scm files isn't for free anymore. > > > > The Scheme compilation felt much slower, and for C++ ccache takes away > > a lot of the pain of recompiles. It also appears to be > > single-threaded? I admit not having timed it in detail. > > Yes, GUILE_AUTO_COMPILE=1 is single-threaded which is why it will never > be the default. If we precompile as part of the build, I wonder if we could use a hack like -djob-count to split lilypond into N jobs where each one does a part of the job. https://www.gnu.org/software/guile/manual/html_node/Compilation.html suggests that one can call the compiler explicitly. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
On Sun, Feb 27, 2022 at 10:39 AM Luca Fascione wrote: > In other words, is it approximately true that "for (almost) any real-life > score the total compilation time > is proportional to the number of NoteHeads, past a certain size"? > I'm guessing you need a few pages worth of material to kill away constant > overheads, but beyond that > is it true that if you double the source size you double the compilation time? it should be, but we have rather complicated page breaking code that is so hairy that nobody understands it fully. I'm not sure there is NP-complete snake in the proverbial grass there. >> I am not sure what you mean by effort spent >> in "preparing" callbacks, could you elaborate? > > > Imagine the grob is a C++ object that remains opaque to Scheme. So it's a > thing for which > in Scheme you move around a blind pointer, but to read property Y-offset > you'd call (grob-get 'Y-offset) > and grob-get is C++. > .. accessing data (eg. offsets): * on first access, the callback gets executed. This is just evaluating ( ). * on second access, the value is cached in an alist, and looking it up is extremely cheap. > Code following a simple pattern like this, once compiled, will largely be > dominated by the > scripting language runtime overhead >From the outside this may seem plausible, but I doubt your intuition here: * the callback tables are also alists. They are cheap (they aren't sped up if you swap them for hash tables) * Scheme has no marshaling: objects are either direct (scheme -> C++ is bit twiddling), or they are indirect (a tagged pair with a C++ pointer in the cdr) IMO The real problem is that we don't have good tooling to see what is going on in the Scheme side of things: C++ has decent perf analysis, but the Guile side of things just looks like a lot of time spent in scm_eval(). Some of it is overhead, some if it might be a poorly implemented Scheme function (which is often easier to fix than reducing overhead.) > (*) 'boxing' is a C# word to mean wrap a POD value in an object to move it > around in a way > that is uniform with the other objects in the language. C# people need to > keep boxing and > unboxing costs in their head in code with lots of PODs being moved around, > because that > cost can dominate the execution of their code. I'm not sure what word is used > in Guile for this > concept. For integers, floats and booleans, the CPU native representation involves some bit operations. Intermediate conversions could be short-cut if you have full type information, and have a sequence of those in a row. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
On Sun, Feb 27, 2022 at 12:31 AM Jean Abou Samra wrote: > > I never said I don't want to fix Guile 2.2 bugs, and you should know > > as I spent lots and lots of time debugging #6218. I also said I > > support moving CI to 2.2, so any MR would pass against 2.2. > > > > I am just asking to not drop 1.8 support. > > > > Most of the work we do isn't actually about Guile anyway, > > > > $ git log --since 2022/01/01 | grep ^commit|wc > > 257 514 12336 > > $ git log --grep=[Gg]uile --since 2022/01/01 | grep ^commit|wc > >7 14 336 > > I agree that the version-specific code we have (cond-expand > and GUILEV2) isn't all that much. On the other hand, I would > be glad to be able to use Guile 2 features, such as Unicode > support, when and unless, (srfi srfi-71) (let and let* > accepting multiple values), S-expression comments, > scm_c_bind_keyword_arguments, and a few others. Now > that my principal concern with the sustainability of > Guile 2 binaries (shipping bytecode with them) is cleared, > I have mixed feelings about when to leave Guile 1 behind. You say you have mixed feelings, but I think (with your updates to compilation), those feelings are ever less mixed? > In my previous post I showed that compiling all Scheme > files can be done in 20s with Guile 2 (so a few seconds > for a single file), and 4s with Guile 3 (so near instant > for one file). Would that address your concern with > compilation speed? Thanks a ton for your investigation into this. This is a game changer: MSDM-reduced 1.8: real 0m14.788s 2.2: real 0m14.648s les-nereides: 1.8: real 0m1.376s 2.2: real 0m1.224s Let's kill 1.8 support. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
On Sat, Feb 26, 2022 at 10:48 PM Jean Abou Samra wrote:
>
> [Jonas]
> > He, I always thought auto-compilation didn't optimize! now don't
> > tell me Guile also applies optimizations while just reading and
> > supposedly interpreting code...
>
> I don't think it does. At least, you don't usually call eval or
> primitive-eval on code to run it repeatedly, just for one-shot
> code, so I don't see the sense it would make. Also, it seems
> to me that the Guile developers are much more focused on compilation.
>
Randomly, it seems to me my own problem with fingering might possibly bring
up
a counter example pattern (for eval vs compile): when you attach a callback
to something that is evaluated
often (in the case of fingering there is an after-line-break callback,
which I am guessing
runs O(count(Fingering)) ~= O(count(NoteHead)). Isn't this a case of scheme
code coming in from
the .ly file (likely an include ly file) and being run O(N) times?
I can't say I understand lilypond super well, but it doesn't feel like it
would employ often
alogrithms that are superlinear in the NoteHead count (it seems most
activities would be
organized as linear scans), so I'm guessing O(N) is as bad as it gets for a
cursory evaluation?
In other words, is it approximately true that "for (almost) any real-life
score the total compilation time
is proportional to the number of NoteHeads, past a certain size"?
I'm guessing you need a few pages worth of material to kill away constant
overheads, but beyond that
is it true that if you double the source size you double the compilation
time?
The background for this was what David was saying, whether code in .ly
files would be optimized or not.
At a guess, I'd think stuff that is big and runs O(n) times might make some
sense to see if need to optimize.
I am sensitive to your other passage of the nightmare it will be to keep
this stuff around and properly invalidate
it upon changes.
> [ skipping over the part regarding Guile 3, since I don't think it's
> > relevant here ]
>
> Perhaps I should have changed the title, but I do think it
> is relevant -- it gives hope for a state where development
> cycles will be easier. When we want Guile 3 is another question.
> I'm in favor of not making the move to Guile 2 wait more
> than it already has, but I wonder if at some point in this release
> cycle (by which I mean before the next stable release) we will want
> to switch to Guile 3.
>
fwiw, this sounds super reasonable to me.
I was just trying to get orders
> of magnitude (for Guile 3 it's 1m30 vs. 4s, no need for
> precise benchmarks to see which is faster :-).
>
Indeed. I was more thinking that not only the opt/no-opt numbers are close,
but also 18-ish to 19ish seconds are close, it is possible that difference
too
is spurious for some reason (I guess I'm saying: there's a possibility you
have been a little lucky or a little unlucky, and your actual runtime
difference is
closer to 2% or maybe 15%, but you happened to take samples at "weird"
times.
I might have some time next week (after 7 march) to run these tests several
times, depends on
some stuff I have going on for next weekend. I'll contact you oob if I do
for some
quick guidance exchange.
I am no performance expert, but LilyPond's performance-critical parts
> are written in C++, so in general I would think the cost of Scheme code
> is spread a bit all over the code base (unlike C++ where you can find
> some rather critical parts taking lots of time, like page breaking,
> or skyline algorithms).
Well but this would be single call from scheme into C++ that take "a long
time",
do I read you right?
Instead I was thinking more of the "dying from a thousand cuts" kinda
scenario.
[continue below]
> I am not sure what you mean by effort spent
> in "preparing" callbacks, could you elaborate?
>
Imagine the grob is a C++ object that remains opaque to Scheme. So it's a
thing for which
in Scheme you move around a blind pointer, but to read property Y-offset
you'd call (grob-get 'Y-offset)
and grob-get is C++.
And then in C++ you'd just have a one-liner accessor that is conceptually
float scheme_grob_get(Grob* grob, sch_value* propname ) {
// typecheck propname to be a symbol
// find out if you have a symbol and what accessor you want to delegate
to
return grob.get_y_offset(); // actual delegation
}
and in turn the getter is something like
inline float get_y_offset() const { return y_offset; /* this is a float
member, say */ }
Code following a simple pattern like this, once compiled, will largely be
dominated by the
scripting language runtime overhead in traversing through the callback
table, finding the
function pointer for the dispatch, marshalling the values from scheme
representation to
something you can mess with in C++ and the way back upon return.
I've read enough of your C++ to see that a lot of this happens in client
code, but either way
it's all code that dominates in cost the execution of the accessor
Re: Blockers for Guile 2.2
Am Sonntag, dem 27.02.2022 um 01:04 +0100 schrieb Han-Wen Nienhuys: > On Sat, Feb 26, 2022 at 2:02 PM Jonas Hahnfeld wrote: > > > while we make work slower for folks that work on large > > > scores and can afford to side-install Guile 1.8. It also makes > > > development slower for ourselves. Yes, that means some of us will > > > develop on a different platform than many users. This has been the > > > case since we started supporting OSX and Windows. > > > > I slightly disagree, running on a different platform is not the same as > > running with a completely different set of dependencies. And we know > > Guile 2.2 is completely different from 1.8. > > > > Everyone who is passionate about Guile 2.2 can develop against Guile 2.2. > > > > So to be clear here: You would release official binaries with Guile 2.2 > > and rely on a subset of developers to fix the bugs while you make it > > clear that you don't want to do that? Why would anybody accept that > > job? > > I never said I don't want to fix Guile 2.2 bugs, and you should know > as I spent lots and lots of time debugging #6218. I also said I > support moving CI to 2.2, so any MR would pass against 2.2. > > I am just asking to not drop 1.8 support. Ok, so let me try to understand the motivation here: It's not for the official binaries, where you want to move away from GUB so it would be Guile 2.2 only. Neither about distributions, which are looking into or have already switched to Guile 2.2. This leaves "power users" that compile LilyPond on their own, with Guile 1.8. Not sure if it's worth complicating our development lives for what will be a transition period anyway. For your local development, if you care about things working with Guile 2.2, you'll have to test at least the final version of a patch with both versions. So I can only assume that you expect a time saving to come from hacking with Guile 1.8 during prototype? > Most of the work we do isn't actually about Guile anyway, > > $ git log --since 2022/01/01 | grep ^commit|wc > 257 514 12336 > $ git log --grep=[Gg]uile --since 2022/01/01 | grep ^commit|wc > 7 14 336 What you're showing is that most commit messages don't mention Guile. IMO the more relevant metric is how many commits touch scm/, in comparison to those that are not purely updates of the documentation: $ git log --oneline --since 2022-01-01 -- flower/ lily/ ly/ scm/ | wc -l 125 $ git log --oneline --since 2022-01-01 -- scm/ | wc -l 52 One third. Since 2021-01-01 that ratio is closer to every second. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Le 27/02/2022 à 01:04, Han-Wen Nienhuys a écrit : On Sat, Feb 26, 2022 at 2:02 PM Jonas Hahnfeld wrote: while we make work slower for folks that work on large scores and can afford to side-install Guile 1.8. It also makes development slower for ourselves. Yes, that means some of us will develop on a different platform than many users. This has been the case since we started supporting OSX and Windows. I slightly disagree, running on a different platform is not the same as running with a completely different set of dependencies. And we know Guile 2.2 is completely different from 1.8. Everyone who is passionate about Guile 2.2 can develop against Guile 2.2. So to be clear here: You would release official binaries with Guile 2.2 and rely on a subset of developers to fix the bugs while you make it clear that you don't want to do that? Why would anybody accept that job? I never said I don't want to fix Guile 2.2 bugs, and you should know as I spent lots and lots of time debugging #6218. I also said I support moving CI to 2.2, so any MR would pass against 2.2. I am just asking to not drop 1.8 support. Most of the work we do isn't actually about Guile anyway, $ git log --since 2022/01/01 | grep ^commit|wc 257 514 12336 $ git log --grep=[Gg]uile --since 2022/01/01 | grep ^commit|wc 7 14 336 I agree that the version-specific code we have (cond-expand and GUILEV2) isn't all that much. On the other hand, I would be glad to be able to use Guile 2 features, such as Unicode support, when and unless, (srfi srfi-71) (let and let* accepting multiple values), S-expression comments, scm_c_bind_keyword_arguments, and a few others. Now that my principal concern with the sustainability of Guile 2 binaries (shipping bytecode with them) is cleared, I have mixed feelings about when to leave Guile 1 behind. I understand the desire to keep fast development cycles, but I think the cognitive burden of wondering whether your code will work on both versions is not desirable in the long run regardless of CI issues. In my previous post I showed that compiling all Scheme files can be done in 20s with Guile 2 (so a few seconds for a single file), and 4s with Guile 3 (so near instant for one file). Would that address your concern with compilation speed? Jean
Re: Blockers for Guile 2.2
On Sat, Feb 26, 2022 at 2:02 PM Jonas Hahnfeld wrote: > > while we make work slower for folks that work on large > > scores and can afford to side-install Guile 1.8. It also makes > > development slower for ourselves. Yes, that means some of us will > > develop on a different platform than many users. This has been the > > case since we started supporting OSX and Windows. > > I slightly disagree, running on a different platform is not the same as > running with a completely different set of dependencies. And we know > Guile 2.2 is completely different from 1.8. > > Everyone who is passionate about Guile 2.2 can develop against Guile 2.2. > > So to be clear here: You would release official binaries with Guile 2.2 > and rely on a subset of developers to fix the bugs while you make it > clear that you don't want to do that? Why would anybody accept that > job? I never said I don't want to fix Guile 2.2 bugs, and you should know as I spent lots and lots of time debugging #6218. I also said I support moving CI to 2.2, so any MR would pass against 2.2. I am just asking to not drop 1.8 support. Most of the work we do isn't actually about Guile anyway, $ git log --since 2022/01/01 | grep ^commit|wc 257 514 12336 $ git log --grep=[Gg]uile --since 2022/01/01 | grep ^commit|wc 7 14 336 -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
[David]
I think where we ultimately want to end up is to have Guile use
optimisation for code loaded from .scm files (which should likely use
byte compilation) while not doing so for Guile code and definitions
invoked from .ly files with # and $ because those more likely than not
are not part of performance-relevant inner loops (and if they do, moving
them to .scm or otherwise specifically marking them might be a
reasonable requirement).
Note that "ultimately" does not mean that this may be the best strategy
right now. But LilyPond documents contain a huge amount of ad-hoc
Scheme code introduced with # (often as simple as a numeric constant or
quoted code that may warrant special-casing, and that actually _is_
special-cased within #{ #} passages already to avoid having to form
closures for it).
For one thing I think we definitely do not want to implement
our own caching of byte-compiled code from .ly files. Because
Guile has no notion of dependency tracking, which means that
if a macro changes, it will not detect that it needs to recompile
its call sites as it will just have expanded the macro and
forgot about the origin of the expanded code. Actually I think
it depends on all bindings from the current module, so it would
be problematic even without macros, though I am not sure about that.
Nevertheless, it would be glaringly user-unfriendly to introduce
caching that does not update automatically for our own user base,
in my opinion.
With that in mind, whether to compile the code before executing
it (thus recompiling at every compilation of the .ly file) or to
simply primitive-eval it is a matter of seeing whether the
byte-compilation is fast enough and whether it actually
enhances usability. Speed should really not be an issue for
code from .ly files (whether those from LilyPond or the users').
On the other hand, byte-compiled code tends to give better
error messages when it fails. I have not experimented yet
(usually I try to do the experiments before I write to the
list, but I'm swamped right now), though I think it shouldn't
be too hard a decision.
The status quo for now is that LilyPond's .scm files are
run by the virtual machine whereas Scheme code in # and $
is run through the evaluator (i.e., primitive-eval). That
is in fact exactly what you describe as "where we ultimately
want to end up".
Of course your timings are quite encouraging here for seeing a path
forward but working out the details of what combinations make best sense
both in the short and the long run is likely going to need quite more
experimentation.
At least, I think the status quo is acceptable. Whether other
strategies can turn out better will be interesting to see,
but can be done in a later step.
[Jonas]
He, I always thought auto-compilation didn't optimize! now don't
tell me Guile also applies optimizations while just reading and
supposedly interpreting code...
I don't think it does. At least, you don't usually call eval or
primitive-eval on code to run it repeatedly, just for one-shot
code, so I don't see the sense it would make. Also, it seems
to me that the Guile developers are much more focused on compilation.
[ skipping over the part regarding Guile 3, since I don't think it's
relevant here ]
Perhaps I should have changed the title, but I do think it
is relevant -- it gives hope for a state where development
cycles will be easier. When we want Guile 3 is another question.
I'm in favor of not making the move to Guile 2 wait more
than it already has, but I wonder if at some point in this release
cycle (by which I mean before the next stable release) we will want
to switch to Guile 3. Of course, that will depend on how easy
it turns out to fix issues like
https://gitlab.com/lilypond/lilypond/-/merge_requests/1230#note_855980027
Yes, it looks like we should do this! On the patch, I think it would be
better to apply the strategy from module/scripts/compile.scm and just
get all available-optimizations from the concatenation of tree-il-
default-optimization-options and cps-default-optimization-options
instead of hard-coding the list.
Yes, that is what should be done if we decide to turn off
them all. I still have to do detailed benchmarking to see
which optimizations might save a few percents and which are
expensive, to decide on the final list. Again, that is just
waiting on me having more time (but of course, speaking to
everyone, feel free to beat me to it and experiment by
yourself).
[Luca]
Jean,
how many times did you run these tests?
Eyeballing your numbers it seems there's effectively no difference in
execution time opt/no-opt and 2.2/3.0.
Is the 5% a stable figure, or is it just a one-sample thing?
It's a one-sample thing. I was just trying to get orders
of magnitude (for Guile 3 it's 1m30 vs. 4s, no need for
precise benchmarks to see which is faster :-).
Would it be a passable inference that the reason the optimizer has
effectively no measurable impact in either
Re: Blockers for Guile 2.2
Jean, how many times did you run these tests? Eyeballing your numbers it seems there's effectively no difference in execution time opt/no-opt and 2.2/3.0. Is the 5% a stable figure, or is it just a one-sample thing? Would it be a passable inference that the reason the optimizer has effectively no measurable impact in either runtime is that the scheme source code runs are comparatively short and contain large amounts of code the optimizer can't change? I'm imagining that if your source is largely an alternation of scheme language built-ins (or scheme library code) interspersed with fairly frequent calls into lilypond's brain, the optimizer can't do much about the latter. At the same time, you might be sitting in front of gains coming from making these API calls more efficient, which could be interesting (albeit largely orthogonal to the discussion at hand). I'm not sure how division of cost works out, if there is overhead in preparing for invoking the callbacks vs executing them, for example. I guess insight in that could help focus effort, if any was warranted. I thought the -O0 compilation time in Guile 3.0 was _really_ cool, I guess it indicates the front-end of the 3.x compiler is vastly more efficient? Seems like it could be an interesting way forward for the dev group to run 3.x with -O0 for iteration cycles, and then do what David is saying to ship the scm file with optimizations on and have the in-score Scheme be just built -O0. Reading briefly the message you posted, it seems -O1 might be a better way to go still, which might regain a teeny tiny bit of speed at a potentially very modest cost (say even if your 3.5s become 4 or 5, you still come out with a net win compared to 2.2's 20s). I agree these results are a very cool finding. L On Sat, Feb 26, 2022 at 2:47 PM Jean Abou Samra wrote: > Le 26/02/2022 à 13:51, Han-Wen Nienhuys a écrit : > > The Scheme compilation felt much slower, and for C++ ccache takes away > > a lot of the pain of recompiles. It also appears to be > > single-threaded? I admit not having timed it in detail. > > OK, I have very good news regarding compilation speed. > Tests are done with > > rm -rf out/share/lilypond/current/guile/ && time out/bin/lilypond > <...snip...> > For one thing, Guile's optimization make about zero difference for the > speed of the resulting LilyPond executable. For another, disabling > optimizations in Guile 2 already results in a good speedup (1min > to 20s), and while Guile 3 is even slower than Guile 2 at the default > optimization level (1m30), with optimizations disabled it becomes > near instant (3.5s). > > Guile 3 being far better at compilation speed with zero optimizations > apparently comes from what is described in > > http://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile > > On the other hand, it does look like Guile 3 is a little slower > than Guile 2 on execution time (5%). > > What do you think? > > Jean >
Re: Blockers for Guile 2.2
Am Samstag, dem 26.02.2022 um 14:47 +0100 schrieb Jean Abou Samra: > Le 26/02/2022 à 13:51, Han-Wen Nienhuys a écrit : > > The Scheme compilation felt much slower, and for C++ ccache takes away > > a lot of the pain of recompiles. It also appears to be > > single-threaded? I admit not having timed it in detail. > > OK, I have very good news regarding compilation speed. > Tests are done with > > rm -rf out/share/lilypond/current/guile/ && time out/bin/lilypond > > (I have GUILE_AUTO_COMPILE=1 in my environment.) > > * master and Guile 2 > > real 0m58,877s > user 0m58,773s > sys 0m0,124s > > Execution time on MSDM.ly: > > real 0m18,870s > user 0m18,727s > sys 0m0,470s > > > * Guile 2, with attached patch disabling all optimizations: He, I always thought auto-compilation didn't optimize! now don't tell me Guile also applies optimizations while just reading and supposedly interpreting code... > real 0m16,791s > user 0m16,700s > sys 0m0,056s > > Execution time on MSDM.ly: > > real 0m18,702s > user 0m18,517s > sys 0m0,509s [ skipping over the part regarding Guile 3, since I don't think it's relevant here ] > For one thing, Guile's optimization make about zero difference for the > speed of the resulting LilyPond executable. For another, disabling > optimizations in Guile 2 already results in a good speedup (1min > to 20s), [...]. > > [...] > > What do you think? Yes, it looks like we should do this! On the patch, I think it would be better to apply the strategy from module/scripts/compile.scm and just get all available-optimizations from the concatenation of tree-il- default-optimization-options and cps-default-optimization-options instead of hard-coding the list. Jonas signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Jean Abou Samra writes:
> For one thing, Guile's optimization make about zero difference for the
> speed of the resulting LilyPond executable. For another, disabling
> optimizations in Guile 2 already results in a good speedup (1min
> to 20s), and while Guile 3 is even slower than Guile 2 at the default
> optimization level (1m30), with optimizations disabled it becomes
> near instant (3.5s).
>
> Guile 3 being far better at compilation speed with zero optimizations
> apparently comes from what is described in
>
> http://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile
>
> On the other hand, it does look like Guile 3 is a little slower
> than Guile 2 on execution time (5%).
>
> What do you think?
I think where we ultimately want to end up is to have Guile use
optimisation for code loaded from .scm files (which should likely use
byte compilation) while not doing so for Guile code and definitions
invoked from .ly files with # and $ because those more likely than not
are not part of performance-relevant inner loops (and if they do, moving
them to .scm or otherwise specifically marking them might be a
reasonable requirement).
Note that "ultimately" does not mean that this may be the best strategy
right now. But LilyPond documents contain a huge amount of ad-hoc
Scheme code introduced with # (often as simple as a numeric constant or
quoted code that may warrant special-casing, and that actually _is_
special-cased within #{ #} passages already to avoid having to form
closures for it).
Of course your timings are quite encouraging here for seeing a path
forward but working out the details of what combinations make best sense
both in the short and the long run is likely going to need quite more
experimentation.
--
David Kastrup
Re: Blockers for Guile 2.2
Le 26/02/2022 à 13:51, Han-Wen Nienhuys a écrit :
The Scheme compilation felt much slower, and for C++ ccache takes away
a lot of the pain of recompiles. It also appears to be
single-threaded? I admit not having timed it in detail.
OK, I have very good news regarding compilation speed.
Tests are done with
rm -rf out/share/lilypond/current/guile/ && time out/bin/lilypond
(I have GUILE_AUTO_COMPILE=1 in my environment.)
* master and Guile 2
real 0m58,877s
user 0m58,773s
sys 0m0,124s
Execution time on MSDM.ly:
real 0m18,870s
user 0m18,727s
sys 0m0,470s
* Guile 2, with attached patch disabling all optimizations:
real 0m16,791s
user 0m16,700s
sys 0m0,056s
Execution time on MSDM.ly:
real 0m18,702s
user 0m18,517s
sys 0m0,509s
* Guile 3, with fixes from
https://gitlab.com/lilypond/lilypond/-/merge_requests/1230
real 1m25,302s
user 1m27,644s
sys 0m1,543s
Execution time on MSDM.ly:
real 0m19,533s
user 0m19,268s
sys 0m0,549s
* Guile 3, with https://gitlab.com/lilypond/lilypond/-/merge_requests/1230
plus second attached patch disabling optimizations:
real 0m3,538s
user 0m3,510s
sys 0m0,108s
(!!)
Execution time on MSDM.ly:
real 0m19,571s
user 0m19,387s
sys 0m0,573s
For one thing, Guile's optimization make about zero difference for the
speed of the resulting LilyPond executable. For another, disabling
optimizations in Guile 2 already results in a good speedup (1min
to 20s), and while Guile 3 is even slower than Guile 2 at the default
optimization level (1m30), with optimizations disabled it becomes
near instant (3.5s).
Guile 3 being far better at compilation speed with zero optimizations
apparently comes from what is described in
http://wingolog.org/archives/2020/06/03/a-baseline-compiler-for-guile
On the other hand, it does look like Guile 3 is a little slower
than Guile 2 on execution time (5%).
What do you think?
Jean
From 864143100d232e663b8f0aa8e236b9187d1a48bc Mon Sep 17 00:00:00 2001
From: Jean Abou Samra
Date: Sat, 26 Feb 2022 13:43:37 +0100
Subject: [PATCH] Disable optimizations (Guile 2)
---
lily/guile-init.cc | 18 ++
lily/include/lily-imports.hh | 3 +++
lily/lily-imports.cc | 3 +++
3 files changed, 24 insertions(+)
diff --git a/lily/guile-init.cc b/lily/guile-init.cc
index 2dfae879e4..8d9d1d4bcf 100644
--- a/lily/guile-init.cc
+++ b/lily/guile-init.cc
@@ -59,6 +59,24 @@ ly_init_ly_module ()
for (vsize i = scm_init_funcs_->size (); i--;)
(scm_init_funcs_->at (i)) ();
+#if GUILEV2
+ SCM opts = SCM_EOL;
+ for (const char *name : {
+"precolor-calls?", "rotate-loops?", "licm?", "specialize-numbers?",
+ "resolve-self-references?", "type-fold?", "cse?", "peel-loops?",
+ "prune-bailouts?", "elide-values?", "specialize-primcalls?", "inline-constructors?",
+ "contify?", "prune-top-level-scopes?", "eliminate-dead-code?", "simplify?",
+ "partial-eval?"
+ })
+{
+ SCM kwd = scm_symbol_to_keyword (ly_symbol2scm (name));
+ opts = scm_cons (kwd, scm_cons (SCM_BOOL_F, opts));
+}
+
+ Guile_user::f_auto_compilation_options = opts;
+#endif
+
+
Cpu_timer timer;
if (is_loglevel (LOG_DEBUG))
{
diff --git a/lily/include/lily-imports.hh b/lily/include/lily-imports.hh
index 6b4ad24880..00d2bd5933 100644
--- a/lily/include/lily-imports.hh
+++ b/lily/include/lily-imports.hh
@@ -28,6 +28,9 @@ extern Scm_module module;
typedef Module_variable Variable;
extern Variable apply;
+#if GUILEV2
+extern Variable f_auto_compilation_options;
+#endif
extern Variable equal;
#if GUILEV2
extern Variable f_default_port_encoding;
diff --git a/lily/lily-imports.cc b/lily/lily-imports.cc
index ca4a3aa015..09d4abe6d4 100644
--- a/lily/lily-imports.cc
+++ b/lily/lily-imports.cc
@@ -24,6 +24,9 @@ namespace Guile_user
Scm_module module ("guile-user");
Variable apply ("apply");
+#if GUILEV2
+Variable f_auto_compilation_options ("%auto-compilation-options");
+#endif
Variable equal ("=");
Variable less ("<");
Variable plus ("+");
--
2.32.0
From 0852cc6277a911da4237b2bf013a45988db8b5d0 Mon Sep 17 00:00:00 2001
From: Jean Abou Samra
Date: Sat, 26 Feb 2022 14:20:37 +0100
Subject: [PATCH] Guile 3: set optimization level to 0
---
lily/guile-init.cc | 3 +++
lily/include/lily-imports.hh | 8
lily/lily-imports.cc | 7 +++
3 files changed, 18 insertions(+)
diff --git a/lily/guile-init.cc b/lily/guile-init.cc
index 2dfae879e4..641d0fc657 100644
--- a/lily/guile-init.cc
+++ b/lily/guile-init.cc
@@ -59,6 +59,8 @@ ly_init_ly_module ()
for (vsize i = scm_init_funcs_->size (); i--;)
(scm_init_funcs_->at (i)) ();
+ Compile::default_optimization_level (to_scm (0));
+
Cpu_timer timer;
if (is_loglevel (LOG_DEBUG))
{
@@ -78,6 +80,7 @@ void
ly_c_init_guile ()
{
Guile_user::module.import ();
+ Compile::module.import ();
Lily::module.boot (ly_init_ly_module);
Re: Blockers for Guile 2.2
Am Samstag, dem 26.02.2022 um 13:51 +0100 schrieb Han-Wen Nienhuys: > On Wed, Feb 23, 2022 at 12:36 PM Jonas Hahnfeld wrote: > > > > > * The concern over CI minutes seems like it's the least important: we > > > can buy more computing power (I'm happy to sponsor), and is the > > > duration of CI much of a concern today? > > > > In the past, you've complained the loudest about slow CI builds for > > merging changes... > > A build for 2.2 and 1.8 could be run in parallel, I think? Yes, but it's still going to be slow because we have exactly one very fast runner. > > > I don't think we have to do the doc build across both 1.8 and 2.2, for > > > example. > > > > I think we do because a full doc build provides coverage and stress > > testing that is unachieved by the regression tests. > > The doc build is exactly the same as the regtest: a lot of small files > that are rendered mostly in per-system mode (rather than per page). > It's just more of the same, so you get better coverage for small > windows of time where objects aren't GC protected. > > Also, you make it seem as if perfect coverage is the only alternative > to scrapping the code altogether. I think that is a false dilemma. > > > > * We're talking about the impact of (the lack of) byte compilation on > > > users, but we haven't discussed the impact on ourselves: the startup > > > slowness affects our every debug/edit cycle, and byte compilation is > > > impractical because we change scm files often (by virtue of checking > > > out other branches). > > > > The same happens for C++ files, you also have to recompile. But it's > > true that editing scm files isn't for free anymore. > > The Scheme compilation felt much slower, and for C++ ccache takes away > a lot of the pain of recompiles. It also appears to be > single-threaded? I admit not having timed it in detail. Yes, GUILE_AUTO_COMPILE=1 is single-threaded which is why it will never be the default. > > > If we need to kill 1.8 support because it blocks > > > something important, then so be it, but given the impact on lilypond > > > development, I'd try to postpone it if possible. > > > > So, you propose that development continues on Guile 1.8, but we > > advertise that we consider Guile 2.2 ready for production use? I don't > > think that makes for a great user experience if we let them find > > problems... Also "postpone" up to what point? > > If we drop 1.8 support now, we can drop just a tiny sliver of > complexity (places marked GUILEV2 and guile-2 in C++ and Scheme > respectively), I think you're underestimating the gain here. > while we make work slower for folks that work on large > scores and can afford to side-install Guile 1.8. It also makes > development slower for ourselves. Yes, that means some of us will > develop on a different platform than many users. This has been the > case since we started supporting OSX and Windows. I slightly disagree, running on a different platform is not the same as running with a completely different set of dependencies. And we know Guile 2.2 is completely different from 1.8. > Everyone who is passionate about Guile 2.2 can develop against Guile 2.2. So to be clear here: You would release official binaries with Guile 2.2 and rely on a subset of developers to fix the bugs while you make it clear that you don't want to do that? Why would anybody accept that job? signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
On Wed, Feb 23, 2022 at 12:36 PM Jonas Hahnfeld wrote: > > * I'm worried that introducing a new version of lilypond that is > > significantly slower than older versions creates an incentive for > > users to stay on older versions. > > > > * I grepped our source code for "guile-2" (scm) and GUILEV2, but the > > divergence of code paths seems pretty minor. Sure, it's inconvenient > > to have the odd conditional or limit ourselves to what is both in 1.8 > > and 2.2, but I'd rather see us drop 1.8 support once we see an > > obstacle that we cannot paper over. > > The problem is, in order to "see an obstacle", we always have to invest > to test every single change with Guile 1.8 and 2.2 during development, > which will be a significant time sink. > > > * The concern over CI minutes seems like it's the least important: we > > can buy more computing power (I'm happy to sponsor), and is the > > duration of CI much of a concern today? > > In the past, you've complained the loudest about slow CI builds for > merging changes... A build for 2.2 and 1.8 could be run in parallel, I think? > > I don't think we have to do the doc build across both 1.8 and 2.2, for > > example. > > I think we do because a full doc build provides coverage and stress > testing that is unachieved by the regression tests. The doc build is exactly the same as the regtest: a lot of small files that are rendered mostly in per-system mode (rather than per page). It's just more of the same, so you get better coverage for small windows of time where objects aren't GC protected. Also, you make it seem as if perfect coverage is the only alternative to scrapping the code altogether. I think that is a false dilemma. > > * We're talking about the impact of (the lack of) byte compilation on > > users, but we haven't discussed the impact on ourselves: the startup > > slowness affects our every debug/edit cycle, and byte compilation is > > impractical because we change scm files often (by virtue of checking > > out other branches). > > The same happens for C++ files, you also have to recompile. But it's > true that editing scm files isn't for free anymore. The Scheme compilation felt much slower, and for C++ ccache takes away a lot of the pain of recompiles. It also appears to be single-threaded? I admit not having timed it in detail. > > If we need to kill 1.8 support because it blocks > > something important, then so be it, but given the impact on lilypond > > development, I'd try to postpone it if possible. > > So, you propose that development continues on Guile 1.8, but we > advertise that we consider Guile 2.2 ready for production use? I don't > think that makes for a great user experience if we let them find > problems... Also "postpone" up to what point? If we drop 1.8 support now, we can drop just a tiny sliver of complexity (places marked GUILEV2 and guile-2 in C++ and Scheme respectively), while we make work slower for folks that work on large scores and can afford to side-install Guile 1.8. It also makes development slower for ourselves. Yes, that means some of us will develop on a different platform than many users. This has been the case since we started supporting OSX and Windows. Everyone who is passionate about Guile 2.2 can develop against Guile 2.2. I'm not opposed to dropping 1.8, as long as we get something substantial in return for it. For example, we might be able to drop all of the GC marking code if we defer to BDWGC for all memory management. That is an attractive proposition, but I'd like to see a prototype before we actually go there. > > * In the discussion, we've been treating "keeping GUB alive" and > > "supporting GUILE 1.8" as equivalent, but is that really the case? We > > can't have GUILE 1.8 for 64-bit windows, but for OSX/Linux, it would > > be an option with the new release scripts too? > > No, Guile 1.8 has mandatory shared libraries for some srfi modules. > There are tricks you can play to make it work with an otherwise static > build, but they are really ugly and certainly not something I think > should be done for production. ack. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
Le 25/02/2022 à 08:41, Jean Abou Samra a écrit : Le 25/02/2022 à 08:18, Jonas Hahnfeld a écrit : If so, that clears my concerns. Just to be sure, can you be more precise here? Does it also clear the concerns about entirely dropping the code for Guile 1.8? If it turns out to work on Windows, it clears that category of concern, yes. The other concern I had Uh, I accidentally deleted a line. The other concern I had is unrelated, it's about performance on macOS *with* bytecode, see the start of the thread. But I'm not sure yet what this is caused by, if it's fixable, ... In summary, more investigation is needed. The Guile statprof output he posted shows a dramatic increase in the time for apply-smob/1, so if I understand it correctly it seems like a slowdown on the C++ side in a translator method (?). With Thomas Scharkowski we communicated off-list and managed to get a profile with the "sample" command, but I had forgotten that these binaries didn't have debugging symbols. That's as far as I got for now. Jean
Re: Blockers for Guile 2.2
Le 25/02/2022 à 08:18, Jonas Hahnfeld a écrit : If so, that clears my concerns. Just to be sure, can you be more precise here? Does it also clear the concerns about entirely dropping the code for Guile 1.8? If it turns out to work on Windows, it clears that category of concern, yes. The other concern I had not sure yet what this is caused by, if it's fixable, ... In summary, more investigation is needed. The Guile statprof output he posted shows a dramatic increase in the time for apply-smob/1, so if I understand it correctly it seems like a slowdown on the C++ side in a translator method (?). With Thomas Scharkowski we communicated off-list and managed to get a profile with the "sample" command, but I had forgotten that these binaries didn't have debugging symbols. That's as far as I got for now. Jean
Re: Blockers for Guile 2.2
Am Freitag, dem 25.02.2022 um 00:08 +0100 schrieb Jean Abou Samra: > > > I read over this thread, but I don't understand what you mean by > > > "downstreams" here. > > > > In my understanding, it's about "downstreams" packaging LilyPond, > > including Linux distributions and parties like HomeBrew and MacPorts. > > But please ask Jean what exactly is required now: > > https://lists.gnu.org/archive/html/lilypond-devel/2022-02/msg00123.html > > GUILE_AUTO_COMPILE=1 is not just for us. The user can have it set > in their environment as a system configuration for all software that > happens to use Guile as well. In my opinion, this is highly unlikely. If nothing is set, auto compilation is the default, only LilyPond chooses to soft-default to disable it (meaning it can be overwritten by explicitly setting the environment variable). I would doubt that anybody except you added to their environment by default. If they have, I maintain that it's their problem. There are far easier ways to shoot yourself into the foot. > > See https://gitlab.com/hahnjo/lilypond/-/commits/guile2-bytecode Let me > > know if this is miraculously sufficient to make people happy and I can > > open a merge request. > > I was going to write a longer reply before this came, but > this mostly obviates it. Am I understanding it correctly that > you can copy the bytecode from Linux binaries to cross-compiled > MingW ones even if this is not automated for now? That is my understanding and the claim I'm making since last year, but I haven't tested this yet. > If so, that clears my concerns. Just to be sure, can you be more precise here? Does it also clear the concerns about entirely dropping the code for Guile 1.8? signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
[Han-Wen] On Tue, Feb 22, 2022 at 12:05 PM Jean Abou Samra wrote: Friends, I don't see this thread coming to a conclusion if it stays between the same three people, and the topic is somewhat important to LilyPond's future. More voices would be helpful. Here are my thoughts: Thanks for sharing your thoughts. The quote is actually from an off-list message BBCed to you, but never mind. [Jonas] My understanding of Guile's development is that certain developers have ideas in their mind, and they eventually end up fully implemented in the repository. There is no such thing as (public) plans. My understanding of Guile's development is that there is one developer, Andy Wingo, who has lots of fun experimenting with compilers, then Ludovic Courtès, who appears more receptive to our needs but more focused on Guix and lacking time for Guile, and a number other contributors who cannot participate very seriously because patches stall for months or years without getting looked at. Guile is being developed in solo. Andy Wingo never writes to mailing lists except to announce releases, does not appear to read neither patches, nor bug reports, nor proposals, and criticism like https://lists.gnu.org/archive/html/guile-devel/2022-02/msg00031.html does not publicly look taken into account either. So the bus factor of Guile is essentially 1 and as long as this will be the model, we will continue to see more funny Scheme compiler optimizations, less C accessibility, and less usability. An good example is https://lists.gnu.org/archive/html/guile-devel/2022-02/msg00030.html Now, even without macros, you can get mysterious issues related to bytecode having been compiled for an outdated version of another file, if you naively use the default optimization level. (And all in a patch release.) Obviously I am not exactly optimistic about Guile's future, and even less about the attractivity of newer Guile versions for LilyPond's use case. By the way, have you watched https://archive.fosdem.org/2020/schedule/event/guile2020/ That is an interesting insight into how Guile evolves. Especially the recurring debate about "Is it OK to lose users?". Anyway. I read over this thread, but I don't understand what you mean by "downstreams" here. In my understanding, it's about "downstreams" packaging LilyPond, including Linux distributions and parties like HomeBrew and MacPorts. But please ask Jean what exactly is required now: https://lists.gnu.org/archive/html/lilypond-devel/2022-02/msg00123.html GUILE_AUTO_COMPILE=1 is not just for us. The user can have it set in their environment as a system configuration for all software that happens to use Guile as well. If that causes LilyPond to try to let Guile write bytecode in a directory where they don't have write access, things will go wrong. (I seem to remember that this would not be an issue with HomeBrew because it lets the user chown its "Cellar" but I am not sure and don't have macOS to test.) That's the sort of thing I was referring to. Again, I don't think it should block starting to make releases exclusively with Guile 2. What I am uncomfortable with is actively removing Guile 1-specific code before this is sorted out. Seehttps://gitlab.com/hahnjo/lilypond/-/commits/guile2-bytecode Let me know if this is miraculously sufficient to make people happy and I can open a merge request. I was going to write a longer reply before this came, but this mostly obviates it. Am I understanding it correctly that you can copy the bytecode from Linux binaries to cross-compiled MingW ones even if this is not automated for now? If so, that clears my concerns. Jean
Re: Blockers for Guile 2.2
[There is much to reply to so I'll make several emails.] Hi Luca, Le 24/02/2022 à 09:13, Luca Fascione a écrit : In case it's useful, I'll share my impressions as a recent addition to this group. I have some experience with rolling out software, gathered in a different field. Where I come from we release often (I think we've averaged in the 30+ cuts per year, roughly 2 every 3 weeks), and our users have tolerance for the occasional rollback, and they roll forward relatively happily. One very important difference is that in that environment it is built into the infrastructure a solid mechanism for picking the software version per project, so that you could be working on several different versions of the software in the same day or so that were transparently tracked for you to stay on their corresponding working files. It seems to me the people that I hear speaking (maybe with the exception of HanWen) have a low level of confidence in the ability of the regression/test suite to provide adequate coverage of the scenarios needed by your current users. (Otherwise I don't even know why this is a discussion in the first place) I would consider this your highest priority: trusting your tests is absolutely crucial. A release going out that passes tests and fails in the field should be brought to be a "0 probability" event. This won't ever be the case for real, hence the occasional rollback: when user X rolls back, a test is made and added to the suite so this won't happen again (preaching to the choir, I get it). I wouldn't give a second of thought to folks rolling back a binary as long as a) you're proactive in saying clearly that there's be a problem and here's how you fix it (roll back, and roll forward once you get the next binary, say); b) this happens rarely, obviously. Guile 2.2 has been under daily by individual developers and some power users for some time now. The problem is more with different operating systems, architectures and setups. Pretty much all developers are running some flavor of GNU/Linux with relatively up-to-date software. The issues that popped up on the release of 2.23.6 (see https://lists.gnu.org/archive/html/lilypond-user/2022-02/msg00155.html) are mostly related to different OSes, OS versions, and locales. Other matrix dimensions we have are the versions of GhostScript, FontConfig, Pango and FreeType. The problem is not the test suite, but the variations in setup. The other aspect that I observe reading this exchange is that nobody is talking about Scheme source in client code. I see you discussing how to pick up the differences between Guile 1.8 and 2.2 in the shipped source, but I don't recall seeing a discussion focusing on user-side changes. And in fact, this is a similar point to the previous one: either these differences are there and are material or they are not. In the second case, you should go ahead and move to 2.2. In fact I'd think you ought to ship current-ish (read non 2.2-specific) scheme source (with bugfixes) with the new runtime, and let that soak in userland for a bit (say 2.24 is that, and then 2.26 contains 2.2 specific source). As I read some people saying certain distributions are already shipping binaries based on Guile 2.2, I suspect maybe 2.22 (on those distributions) is a passable proxy for this stage. In the first case (Guile 1.8 vs 2.2 actually makes a real difference in source semantics and correctness(*), forget about the speed) your first concern must be what to do about users migrating their source and what cost will that be, in terms of time investment, instabilities or defects introduced and all that, which will dominate your ability to reduce your maintenance exposure making sure the majority of your user base will in fact pick up the new release line. This is not too much of an issue. Guile 2 was a big change on the "infrastructure" side: a bytecode compiler in addition to the normal interpreter, which puts requirements on the code for being used (the normal interpreter still works but is vastly slower), a new garbage collector, relevant on the C (for us C++) side, Unicode support. Have I missed something? I think these are the outstanding points. Scheme code in LilyPond files is not precompiled, so it doesn't have to make the byte compiler happy; it doesn't need to care about the underlying GC; and Unicode support, of which the poor performance has been an issue for the scale of LilyPond, should not be an issue performance-wise for typical Scheme code and will more often than not just make code work with special characters that didn't before. The main differences for Scheme code are the deprecation of omitting the port argument to format (just prints a warning, still works), the move of curried definitions to a module (not problematic in LilyPond since we include it by default), and the impossibility to use definitions in the body of a cond form (easily worked around with let). Again, I may
Re: Blockers for Guile 2.2
Am Donnerstag, dem 24.02.2022 um 18:25 +0100 schrieb Luca Fascione: > On Thu, Feb 24, 2022 at 5:44 PM Jonas Hahnfeld > wrote: > > I will not reply to most of your message; I suspect that your > > experience comes from a corporate environment where people are paid > > full time to work on software. > > It does, yes. > > > In my opinion, many of the points are simply not relevant in a > > relatively small community of volunteers, for example the release > > frequency (which is already quite high for an open source project I > > would say). Same for testing, in particular all possible > > combinations of dependencies. > > Forgive me Jonas, I don't follow. It seems to me any means to > improve the quality of the software development process would be fair > game, no? A better process means all the folks involved maximize the > ratio of effort into the project to results coming out into the hands > of users, I don't understand why this would be undesirable to any > community of engineers no matter whether they write this code because > it's their job or their passion after hours. If anything, I'd think > the folks that do it after hours would be the most interested in > focusing on the fun side and not on the "fix the boring bugs and > regressions" side, no? Exactly, and making a new release every 10 days is certainly not on the top of my "fun side". Neither is extending the test suite in order for it to reliably catch all possible regressions that could arise from supporting Guile 1.8 and 2.2 in parallel. > > > Given the switch to 2.2 hasn't happened yet, and as I am reading > > > through these emails, it has been a long process, wouldn't moving > > > to 3.0 instead be a better way to capitalize on the effort and > > > push out the next round of this level of pain to a later date? > > > > The question is, would this make things better? Jumping across even > > more versions certainly doesn't promise to be an easier transition. > > I guess I was looking at it from the opposite end: imagine it makes > _this_ transition a little harder, but then postpones the next one > (which will inevitably come) by several years. Wouldn't this be an > interesting deal for your users? (and engineers, of course) > > I guess I'm saying: say that in 2025 you'll have to go to Guile 3 > anyways (maybe because 2.2.x goes unsupported or whatever reason). > What tells us that change will be easier than this? Testing. You can already run with 3.0 if you really want to. Not everything works and you get drowned by warnings, but the changes will be smaller than from 1.8 to 2.2. What would be more difficult, though, is making LilyPond with *three* versions of Guile in the same code. > On the other hand, if you were to adopt 3 today, you can gradually > upgrade it as you go and try and stay on the 3.x train (maybe you do > a tick/tock thing where you pull guile up every other version bump, > as a random example). Or anyways be on a supported runtime for a few > years longer. I expect that support for Guile 3.0 will follow after we drop 1.8 and only support version 2.2, and it will be relatively smooth. That said, requiring this to happen now will delay the transition away from GUB and Guile 1.8 even longer. > Now of course, if instead of this being a "little" harder, it is in > fact a "lot" harder, none of what I'm saying makes any sense. > I don't know the specifics, but I know the pattern can reduce effort > and improve the "fun" to "drudge" ratios, when the conditions are > right. I don't agree. Projects are the most fun if they are clearly scoped and don't drag on "forever". We clearly failed that objective already, so why keep extending? Jonas signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Jonas Hahnfeld via Discussions on LilyPond development writes: > Am Donnerstag, dem 24.02.2022 um 09:13 +0100 schrieb Luca Fascione: >> Last thought: as I am currently learning Scheme and Guile, and I >> noticed 3.0.x has been out for a couple years now and seems to be >> benchmarking with speeds comparable to the 1.8.x line (according to >> their release notes). > > It's important to differentiate between their benchmarks and the real- > world impact on a complex project like LilyPond. There have been > preliminary tests and they indicate that it's still a lot slower than > Guile 1.8 without bytecode. Whether it will be faster than Guile 2.2 > for our use-cases afterwards, I don't know. I seem to remember that 3.0 vs 2.2 is not all that much of a difference: the large jump was 2.0 to 2.2 due to a different byte code machine. -- David Kastrup
Re: Blockers for Guile 2.2
On Thu, Feb 24, 2022 at 5:44 PM Jonas Hahnfeld wrote: > I will not reply to most of your message; I suspect that your > experience comes from a corporate environment where people are paid > full time to work on software. It does, yes. > In my opinion, many of the points are > simply not relevant in a relatively small community of volunteers, for > example the release frequency (which is already quite high for an open > source project I would say). Same for testing, in particular all > possible combinations of dependencies. > Forgive me Jonas, I don't follow. It seems to me any means to improve the quality of the software development process would be fair game, no? A better process means all the folks involved maximize the ratio of effort into the project to results coming out into the hands of users, I don't understand why this would be undesirable to any community of engineers no matter whether they write this code because it's their job or their passion after hours. If anything, I'd think the folks that do it after hours would be the most interested in focusing on the fun side and not on the "fix the boring bugs and regressions" side, no? > The one thing I want to get straight are the comments about Guile 3.0 > because that claim keeps coming up: > > Am Donnerstag, dem 24.02.2022 um 09:13 +0100 schrieb Luca Fascione: > > [...] 3.0.x [...] seems to be benchmarking with speeds comparable to > the 1.8.x line > > It's important to differentiate between their benchmarks and the real- > world impact on a complex project like LilyPond. There have been > preliminary tests and they indicate that it's still a lot slower than > Guile 1.8 without bytecode. Whether it will be faster than Guile 2.2 > for our use-cases afterwards, I don't know. > I see. Cool, I wasn't aware. Great to hear it's already been tested, I just said because nobody was talking about it, and HanWen asked folks about the perspective of the Guile engineers for upcoming evolutions. Seems pretty clear from a casual browse of their page that they don't intend to work on 2.2 any longer, and that their efforts are on 3.x now. > > Given the switch to 2.2 hasn't happened yet, and as I am reading > > through these emails, it has been a long process, wouldn't moving to > > 3.0 instead be a better way to capitalize on the effort and push out > > the next round of this level of pain to a later date? > > The question is, would this make things better? Jumping across even > more versions certainly doesn't promise to be an easier transition. > I guess I was looking at it from the opposite end: imagine it makes _this_ transition a little harder, but then postpones the next one (which will inevitably come) by several years. Wouldn't this be an interesting deal for your users? (and engineers, of course) I guess I'm saying: say that in 2025 you'll have to go to Guile 3 anyways (maybe because 2.2.x goes unsupported or whatever reason). What tells us that change will be easier than this? On the other hand, if you were to adopt 3 today, you can gradually upgrade it as you go and try and stay on the 3.x train (maybe you do a tick/tock thing where you pull guile up every other version bump, as a random example). Or anyways be on a supported runtime for a few years longer. Now of course, if instead of this being a "little" harder, it is in fact a "lot" harder, none of what I'm saying makes any sense. I don't know the specifics, but I know the pattern can reduce effort and improve the "fun" to "drudge" ratios, when the conditions are right. Cheers, L >
Re: Blockers for Guile 2.2
Hi, I will not reply to most of your message; I suspect that your experience comes from a corporate environment where people are paid full time to work on software. In my opinion, many of the points are simply not relevant in a relatively small community of volunteers, for example the release frequency (which is already quite high for an open source project I would say). Same for testing, in particular all possible combinations of dependencies. The one thing I want to get straight are the comments about Guile 3.0 because that claim keeps coming up: Am Donnerstag, dem 24.02.2022 um 09:13 +0100 schrieb Luca Fascione: > Last thought: as I am currently learning Scheme and Guile, and I > noticed 3.0.x has been out for a couple years now and seems to be > benchmarking with speeds comparable to the 1.8.x line (according to > their release notes). It's important to differentiate between their benchmarks and the real- world impact on a complex project like LilyPond. There have been preliminary tests and they indicate that it's still a lot slower than Guile 1.8 without bytecode. Whether it will be faster than Guile 2.2 for our use-cases afterwards, I don't know. > Given the switch to 2.2 hasn't happened yet, and as I am reading > through these emails, it has been a long process, wouldn't moving to > 3.0 instead be a better way to capitalize on the effort and push out > the next round of this level of pain to a later date? The question is, would this make things better? Jumping across even more versions certainly doesn't promise to be an easier transition. Jonas signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
In case it's useful, I'll share my impressions as a recent addition to this group. I have some experience with rolling out software, gathered in a different field. Where I come from we release often (I think we've averaged in the 30+ cuts per year, roughly 2 every 3 weeks), and our users have tolerance for the occasional rollback, and they roll forward relatively happily. One very important difference is that in that environment it is built into the infrastructure a solid mechanism for picking the software version per project, so that you could be working on several different versions of the software in the same day or so that were transparently tracked for you to stay on their corresponding working files. It seems to me the people that I hear speaking (maybe with the exception of HanWen) have a low level of confidence in the ability of the regression/test suite to provide adequate coverage of the scenarios needed by your current users. (Otherwise I don't even know why this is a discussion in the first place) I would consider this your highest priority: trusting your tests is absolutely crucial. A release going out that passes tests and fails in the field should be brought to be a "0 probability" event. This won't ever be the case for real, hence the occasional rollback: when user X rolls back, a test is made and added to the suite so this won't happen again (preaching to the choir, I get it). I wouldn't give a second of thought to folks rolling back a binary as long as a) you're proactive in saying clearly that there's be a problem and here's how you fix it (roll back, and roll forward once you get the next binary, say); b) this happens rarely, obviously. The other aspect that I observe reading this exchange is that nobody is talking about Scheme source in client code. I see you discussing how to pick up the differences between Guile 1.8 and 2.2 in the shipped source, but I don't recall seeing a discussion focusing on user-side changes. And in fact, this is a similar point to the previous one: either these differences are there and are material or they are not. In the second case, you should go ahead and move to 2.2. In fact I'd think you ought to ship current-ish (read non 2.2-specific) scheme source (with bugfixes) with the new runtime, and let that soak in userland for a bit (say 2.24 is that, and then 2.26 contains 2.2 specific source). As I read some people saying certain distributions are already shipping binaries based on Guile 2.2, I suspect maybe 2.22 (on those distributions) is a passable proxy for this stage. In the first case (Guile 1.8 vs 2.2 actually makes a real difference in source semantics and correctness(*), forget about the speed) your first concern must be what to do about users migrating their source and what cost will that be, in terms of time investment, instabilities or defects introduced and all that, which will dominate your ability to reduce your maintenance exposure making sure the majority of your user base will in fact pick up the new release line. (*) not that this seems to me a very realistic thing for the Guile folks to do intentionally Last thought: as I am currently learning Scheme and Guile, and I noticed 3.0.x has been out for a couple years now and seems to be benchmarking with speeds comparable to the 1.8.x line (according to their release notes). Given the switch to 2.2 hasn't happened yet, and as I am reading through these emails, it has been a long process, wouldn't moving to 3.0 instead be a better way to capitalize on the effort and push out the next round of this level of pain to a later date? (Again, I am expecting this has been discussed before, but still occasionally it's good to re-evaluate decisions: circumstances change, new evidence comes up, time goes by) I do realize this will stir many folks, and that this subject is understandably very sensitive, but it's possible that a pause and refocusing could be of use. I feel a lot of drive and energy and dedication to the project from all of you, and it seems very clear to me the folks that are writing maintain very high ethical standard and truly want the best for the project. This is the reason why I joined lilypond-devel btw, the people are awesome. (and I'll admit I also think the program is pretty cool ;) ). However, I also think I observe you all having a difficult time trusting that things will go well, there seems to be a high level of fear that "something bad" will happen. If I may propose a perspective: don't have fear of that, because something will break for sure. Nobody can avoid that, but what you can do, and what will make a positive difference to your users is your ability to help them through the stumble, support them in their difficulties, and keep moving ahead together. Thanks for your time, hopefully some of these thoughts can be of use L
Re: Blockers for Guile 2.2
Am Mittwoch, dem 23.02.2022 um 16:37 +0100 schrieb David Kastrup: > Jonas Hahnfeld via Discussions on LilyPond development > writes: > > > > If we need to kill 1.8 support because it blocks something important, > > > then so be it, but given the impact on lilypond development, I'd try > > > to postpone it if possible. > > > > So, you propose that development continues on Guile 1.8, but we > > advertise that we consider Guile 2.2 ready for production use? > > We advertise that Guile 2.2 is considered to be workable. And what version would the official binaries use? Remain the sole user of Guile 1.8 while Linux distributions and macOS package managers switch to Guile 2.2? > > I don't think that makes for a great user experience if we let them > > find problems... > > Uh, that's the whole point of having unstable releases, isn't it? Maybe, but so far we at least tested the combination of dependency versions during development, even before an unstable release happens. That would be gone and only a brave subgroup of developers testing the integration with Guile 2.2 could notice problems before they pop up for users. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
>> [First of all: Thanks, Jonas, for releasing 2.22.2 today!] > > [Well, I didn't do this alone, Phil does most of the actual release > procedure like editing the right files that both of us keep > forgetting and things like that] D'oh, sorry for that omission, Phil – many cudos to you, too! Werner
Re: Blockers for Guile 2.2
> See https://gitlab.com/hahnjo/lilypond/-/commits/guile2-bytecode Let > me know if this is miraculously sufficient to make people happy and > I can open a merge request. For me, this looks very good, thanks. Werner
Re: Blockers for Guile 2.2
Jonas Hahnfeld via Discussions on LilyPond development writes: > Am Mittwoch, dem 23.02.2022 um 12:09 +0100 schrieb Han-Wen Nienhuys: >> * I grepped our source code for "guile-2" (scm) and GUILEV2, but the >> divergence of code paths seems pretty minor. Sure, it's inconvenient >> to have the odd conditional or limit ourselves to what is both in 1.8 >> and 2.2, but I'd rather see us drop 1.8 support once we see an >> obstacle that we cannot paper over. > > The problem is, in order to "see an obstacle", we always have to > invest to test every single change with Guile 1.8 and 2.2 during > development, which will be a significant time sink. If it is an obstacle, it will make itself seen soon enough. >> * The concern over CI minutes seems like it's the least important: we >> can buy more computing power (I'm happy to sponsor), and is the >> duration of CI much of a concern today? > > In the past, you've complained the loudest about slow CI builds for > merging changes... Let's focus on "today". >> If we need to kill 1.8 support because it blocks something important, >> then so be it, but given the impact on lilypond development, I'd try >> to postpone it if possible. > > So, you propose that development continues on Guile 1.8, but we > advertise that we consider Guile 2.2 ready for production use? We advertise that Guile 2.2 is considered to be workable. > I don't think that makes for a great user experience if we let them > find problems... Uh, that's the whole point of having unstable releases, isn't it? > Also "postpone" up to what point? See above regarding "obstacles". >> I'm glad to hear that, but do we know about the GUILE team's plans in >> this space? It would suck if they want to move to CPU dependent >> bytecode. > > My understanding of Guile's development is that certain developers > have ideas in their mind, and they eventually end up fully implemented > in the repository. There is no such thing as (public) plans. And that's the polite version of putting it. -- David Kastrup
Re: Blockers for Guile 2.2
Am Mittwoch, dem 23.02.2022 um 12:09 +0100 schrieb Han-Wen Nienhuys: > On Tue, Feb 22, 2022 at 12:05 PM Jean Abou Samra wrote: > > > > Friends, > > > > I don't see this thread coming to a conclusion if it stays between the same > > three people, and the topic is somewhat important to LilyPond's future. > > More voices would be helpful. > > > Here are my thoughts: > > * GUB needs to die. Really: I don't ever want to touch that code > again, and I can't fault anyone else for feeling the same. > > * We've got GUILE 2.2 support to a state that we can vouch for. We > need to keep that state. This means we should switch CI to run on > GUILE 2.2 (It's "Guile", not GUILE) > * I'm worried that introducing a new version of lilypond that is > significantly slower than older versions creates an incentive for > users to stay on older versions. > > * I grepped our source code for "guile-2" (scm) and GUILEV2, but the > divergence of code paths seems pretty minor. Sure, it's inconvenient > to have the odd conditional or limit ourselves to what is both in 1.8 > and 2.2, but I'd rather see us drop 1.8 support once we see an > obstacle that we cannot paper over. The problem is, in order to "see an obstacle", we always have to invest to test every single change with Guile 1.8 and 2.2 during development, which will be a significant time sink. > * The concern over CI minutes seems like it's the least important: we > can buy more computing power (I'm happy to sponsor), and is the > duration of CI much of a concern today? In the past, you've complained the loudest about slow CI builds for merging changes... > I don't think we have to do the doc build across both 1.8 and 2.2, for > example. I think we do because a full doc build provides coverage and stress testing that is unachieved by the regression tests. > * We're talking about the impact of (the lack of) byte compilation on > users, but we haven't discussed the impact on ourselves: the startup > slowness affects our every debug/edit cycle, and byte compilation is > impractical because we change scm files often (by virtue of checking > out other branches). The same happens for C++ files, you also have to recompile. But it's true that editing scm files isn't for free anymore. > If we need to kill 1.8 support because it blocks > something important, then so be it, but given the impact on lilypond > development, I'd try to postpone it if possible. So, you propose that development continues on Guile 1.8, but we advertise that we consider Guile 2.2 ready for production use? I don't think that makes for a great user experience if we let them find problems... Also "postpone" up to what point? > * In the discussion, we've been treating "keeping GUB alive" and > "supporting GUILE 1.8" as equivalent, but is that really the case? We > can't have GUILE 1.8 for 64-bit windows, but for OSX/Linux, it would > be an option with the new release scripts too? No, Guile 1.8 has mandatory shared libraries for some srfi modules. There are tricks you can play to make it work with an otherwise static build, but they are really ugly and certainly not something I think should be done for production. > > No, it can't. This approach fundamentally assumes that you can run the > > lilypond binary, which is not true in cross-compilation setups. If it's > > about the "with-target", then you've solved a problem that doesn't > > exist: Guile bytecode (at least for version 2.2) only cares about the > > "cpu-endianness" and "triplet-pointer-size". As all relevant CPU > > architectures of today are little-endian, and we only care about 64-bit > > the bytecodes are identical (tested with a simple module, compiled for > > x86_64-pc-linux-gnu, x86_64-w64-mingw32, powerpc64le-pc-linux-gnu). > > I'm glad to hear that, but do we know about the GUILE team's plans in > this space? It would suck if they want to move to CPU dependent > bytecode. My understanding of Guile's development is that certain developers have ideas in their mind, and they eventually end up fully implemented in the repository. There is no such thing as (public) plans. > > properly solve the setup for "downstreams" that apparently is now a > > requirement, but at least doesn't require additional effort. I wrote > > I read over this thread, but I don't understand what you mean by > "downstreams" here. In my understanding, it's about "downstreams" packaging LilyPond, including Linux distributions and parties like HomeBrew and MacPorts. But please ask Jean what exactly is required now: https://lists.gnu.org/archive/html/lilypond-devel/2022-02/msg00123.html signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
On Tue, Feb 22, 2022 at 12:05 PM Jean Abou Samra wrote: > > Friends, > > I don't see this thread coming to a conclusion if it stays between the same > three people, and the topic is somewhat important to LilyPond's future. More > voices would be helpful. Here are my thoughts: * GUB needs to die. Really: I don't ever want to touch that code again, and I can't fault anyone else for feeling the same. * We've got GUILE 2.2 support to a state that we can vouch for. We need to keep that state. This means we should switch CI to run on GUILE 2.2 * I'm worried that introducing a new version of lilypond that is significantly slower than older versions creates an incentive for users to stay on older versions. * I grepped our source code for "guile-2" (scm) and GUILEV2, but the divergence of code paths seems pretty minor. Sure, it's inconvenient to have the odd conditional or limit ourselves to what is both in 1.8 and 2.2, but I'd rather see us drop 1.8 support once we see an obstacle that we cannot paper over. * The concern over CI minutes seems like it's the least important: we can buy more computing power (I'm happy to sponsor), and is the duration of CI much of a concern today? I don't think we have to do the doc build across both 1.8 and 2.2, for example. * We're talking about the impact of (the lack of) byte compilation on users, but we haven't discussed the impact on ourselves: the startup slowness affects our every debug/edit cycle, and byte compilation is impractical because we change scm files often (by virtue of checking out other branches). If we need to kill 1.8 support because it blocks something important, then so be it, but given the impact on lilypond development, I'd try to postpone it if possible. * In the discussion, we've been treating "keeping GUB alive" and "supporting GUILE 1.8" as equivalent, but is that really the case? We can't have GUILE 1.8 for 64-bit windows, but for OSX/Linux, it would be an option with the new release scripts too? > No, it can't. This approach fundamentally assumes that you can run the > lilypond binary, which is not true in cross-compilation setups. If it's > about the "with-target", then you've solved a problem that doesn't > exist: Guile bytecode (at least for version 2.2) only cares about the > "cpu-endianness" and "triplet-pointer-size". As all relevant CPU > architectures of today are little-endian, and we only care about 64-bit > the bytecodes are identical (tested with a simple module, compiled for > x86_64-pc-linux-gnu, x86_64-w64-mingw32, powerpc64le-pc-linux-gnu). I'm glad to hear that, but do we know about the GUILE team's plans in this space? It would suck if they want to move to CPU dependent bytecode. > properly solve the setup for "downstreams" that apparently is now a > requirement, but at least doesn't require additional effort. I wrote I read over this thread, but I don't understand what you mean by "downstreams" here. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: Blockers for Guile 2.2
Am Dienstag, dem 22.02.2022 um 09:02 +0100 schrieb Jonas Hahnfeld via Discussions on LilyPond development: > Am Montag, dem 21.02.2022 um 22:44 +0100 schrieb Jean Abou Samra: > > Hi, > > > > Sorry for the late reply, I hoped to have a merge request > > ready to implement but life is getting in the way. > > Nevertheless, a proof of concept is this: > > > > > > From 95794324cd4a637c4735447b672a1de91416cc4a Mon Sep 17 00:00:00 2001 > > From: Jean Abou Samra > > Date: Sun, 20 Feb 2022 17:00:20 +0100 > > Subject: [PATCH] WIP: allow separate Guile byte-compilation > > > > Benefits: can be integrated into the build system, can byte-compile > > all files even if not used, can byte-cross-compile. > > No, it can't. This approach fundamentally assumes that you can run the > lilypond binary, which is not true in cross-compilation setups. If it's > about the "with-target", then you've solved a problem that doesn't > exist: Guile bytecode (at least for version 2.2) only cares about the > "cpu-endianness" and "triplet-pointer-size". As all relevant CPU > architectures of today are little-endian, and we only care about 64-bit > the bytecodes are identical (tested with a simple module, compiled for > x86_64-pc-linux-gnu, x86_64-w64-mingw32, powerpc64le-pc-linux-gnu). So > in essence, instead of writing this additioanl code, we could just use > GUILE_AUTO_COMPILE=1 and collect the .go files. Which still doesn't > properly solve the setup for "downstreams" that apparently is now a > requirement, but at least doesn't require additional effort. I wrote > this in the discussion last weekend. See https://gitlab.com/hahnjo/lilypond/-/commits/guile2-bytecode Let me know if this is miraculously sufficient to make people happy and I can open a merge request. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Am Dienstag, dem 22.02.2022 um 11:29 -0600 schrieb Karlin High: > On 2/22/2022 10:55 AM, Werner LEMBERG wrote: > > In particular, we can't tell non-developers "Please use the > > current development version, which works very reliably" and introduce > > a severe slowness at the same time. > > Perhaps that advice could be suspended for one series of development > versions? Doing one last Guile 1.8 stable release before the Guile 2.2 > transition, then advising to stick with that if interim slowness in > development versions is unacceptable. I mean, that "last Guile 1.8 stable release" is already there: 2.22.x, with 2.22.2 being the latest incarnation. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Am Dienstag, dem 22.02.2022 um 16:55 + schrieb Werner LEMBERG: > [First of all: Thanks, Jonas, for releasing 2.22.2 today!] [Well, I didn't do this alone, Phil does most of the actual release procedure like editing the right files that both of us keep forgetting and things like that] > I guess most people like me lack the skills to give helpful comments, > so they stay silent. The only part where I can voice an opinion is > the following. > > > > Meanwhile, please take a look at reality: Linux distributions are > > > switching to Guile 2.2 for LilyPond, no matter what the state is > > > and what our documentation recommends. Debian attempted to > > > switch for the stable release 2.22.0 (that is now in Debian > > > stable) and we convinced them to stick to Guile 1.8 more-or-less > > > last minute. Arch Linux switched to using Guile 2.2 twice, but I > > > was able to convince them that they should wait. Gentoo has an > > > option to build with Guile 2, not sure if it's used by default or > > > up to the user. The package in Fedora continues to use Guile 2.2 > > > IIRC. > > > > > > What this tells me is that we get a split user community if we do > > > nothing. If we want to avoid that and pro-actively test and fix > > > issues, we have to switch *now* and lead the transition. > > Jonas, you are doing a wonderful job by working on the Guile 2.2 > transition. However, it seems to me that it will still take a > significant time until we have something that can be used for daily > work. Even if I don't fully agree, what's your take then on the real situation that I described? Isn't that even worse from a user's perspective? > Otherwise I could imagine to do the following. > > (a) Incorporate Guile 1.8 into the LilyPond tarball to produce > releases without fearing that distributions eliminate LilyPond > packages. > > (b) Produce development releases with Guile 1.8 for people in the (1) > group, probably with GUB. Recent MacOS users would have to use > MacPorts. > > (c) Produce development releases with Guile 2.2 for people in the (2) > group with Jonas' build system. > > I'm not sure how much of my suggestion can be automated to reduce > manual work on it. This would basically freeze the current status of doing releases with GUB - while I made sure the binaries with Guile 2.2 can be produced from a tarball, GUB assumes that it's the authoritative release tool. And (a) and (b) makes it necessary to keep GUB alive *and* precludes us from making progress for all the reasons I mentioned on Sunday and yesterday. Jonas signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
I expect this has been considered before, but what is it that makes it unpalatable to have a step like initex for TeX to build the .go files upon installation? Wouldn't it solve the issue at hand? (The portability would be addressed by the fact that it's the target platform to build online, and you'd have a certain tolerance for variability at the destination site that you could mop up with a process like that would be difficult to handle apriori, I guess short of producing several built packages) L On Tue, Feb 22, 2022 at 6:29 PM Karlin High wrote: > On 2/22/2022 10:55 AM, Werner LEMBERG wrote: > > In particular, we can't tell non-developers "Please use the > > current development version, which works very reliably" and introduce > > a severe slowness at the same time. > > Perhaps that advice could be suspended for one series of development > versions? Doing one last Guile 1.8 stable release before the Guile 2.2 > transition, then advising to stick with that if interim slowness in > development versions is unacceptable. > -- > Karlin High > Missouri, USA > >
Re: Blockers for Guile 2.2
On 2/22/2022 10:55 AM, Werner LEMBERG wrote: In particular, we can't tell non-developers "Please use the current development version, which works very reliably" and introduce a severe slowness at the same time. Perhaps that advice could be suspended for one series of development versions? Doing one last Guile 1.8 stable release before the Guile 2.2 transition, then advising to stick with that if interim slowness in development versions is unacceptable. -- Karlin High Missouri, USA
Re: Blockers for Guile 2.2
Hi all, > there are two types of testers of development versions. > > (1) The casual user that needs this and that feature fixed, and who >isn't satisfied with the stable release and curious enough to >update. I think that we are going to lose a significant portion >of people if development versions are much slower for no immediate >benefits. > > (2) The hardcore development team and related friends, who are willing >to bear the slowness, ironing out issues on the way to recent >Guile versions. In certain ways, I'm part of both of those groups. ;) I'll definitely freeze my "power user" version on the one right before any significant slowdown is introduced: my engraving work tends to be mission-critical and time-sensitive, and given my predilection to getting things done at the last possible moment ;) I need all the speed I can get when it comes to score compilation. But I'm happy and comfortable having multiple versions on my system (yay Frescobaldi!), so I'm also able to work on and test the absolute bleeding-edge version(s) when my time and energy [and project load] allow. Cheers, Kieren.
Re: Blockers for Guile 2.2
[First of all: Thanks, Jonas, for releasing 2.22.2 today!] I guess most people like me lack the skills to give helpful comments, so they stay silent. The only part where I can voice an opinion is the following. >> Meanwhile, please take a look at reality: Linux distributions are >> switching to Guile 2.2 for LilyPond, no matter what the state is >> and what our documentation recommends. Debian attempted to switch >> for the stable release 2.22.0 (that is now in Debian stable) and we >> convinced them to stick to Guile 1.8 more-or-less last minute. >> Arch Linux switched to using Guile 2.2 twice, but I was able to >> convince them that they should wait. Gentoo has an option to build >> with Guile 2, not sure if it's used by default or up to the user. >> The package in Fedora continues to use Guile 2.2 IIRC. >> >> What this tells me is that we get a split user community if we do >> nothing. If we want to avoid that and pro-actively test and fix >> issues, we have to switch *now* and lead the transition. Jonas, you are doing a wonderful job by working on the Guile 2.2 transition. However, it seems to me that it will still take a significant time until we have something that can be used for daily work. In particular, we can't tell non-developers "Please use the current development version, which works very reliably" and introduce a severe slowness at the same time. As far as I can see, there are two types of testers of development versions. (1) The casual user that needs this and that feature fixed, and who isn't satisfied with the stable release and curious enough to update. I think that we are going to lose a significant portion of people if development versions are much slower for no immediate benefits. (2) The hardcore development team and related friends, who are willing to bear the slowness, ironing out issues on the way to recent Guile versions. Do we accept the loss of people in the (1) group? These people test everything of LilyPond's changes except Guile. Otherwise I could imagine to do the following. (a) Incorporate Guile 1.8 into the LilyPond tarball to produce releases without fearing that distributions eliminate LilyPond packages. (b) Produce development releases with Guile 1.8 for people in the (1) group, probably with GUB. Recent MacOS users would have to use MacPorts. (c) Produce development releases with Guile 2.2 for people in the (2) group with Jonas' build system. I'm not sure how much of my suggestion can be automated to reduce manual work on it. Werner
Re: Blockers for Guile 2.2
Am Montag, dem 21.02.2022 um 22:44 +0100 schrieb Jean Abou Samra: > Hi, > > Sorry for the late reply, I hoped to have a merge request > ready to implement but life is getting in the way. > Nevertheless, a proof of concept is this: > > > From 95794324cd4a637c4735447b672a1de91416cc4a Mon Sep 17 00:00:00 2001 > From: Jean Abou Samra > Date: Sun, 20 Feb 2022 17:00:20 +0100 > Subject: [PATCH] WIP: allow separate Guile byte-compilation > > Benefits: can be integrated into the build system, can byte-compile > all files even if not used, can byte-cross-compile. No, it can't. This approach fundamentally assumes that you can run the lilypond binary, which is not true in cross-compilation setups. If it's about the "with-target", then you've solved a problem that doesn't exist: Guile bytecode (at least for version 2.2) only cares about the "cpu-endianness" and "triplet-pointer-size". As all relevant CPU architectures of today are little-endian, and we only care about 64-bit the bytecodes are identical (tested with a simple module, compiled for x86_64-pc-linux-gnu, x86_64-w64-mingw32, powerpc64le-pc-linux-gnu). So in essence, instead of writing this additioanl code, we could just use GUILE_AUTO_COMPILE=1 and collect the .go files. Which still doesn't properly solve the setup for "downstreams" that apparently is now a requirement, but at least doesn't require additional effort. I wrote this in the discussion last weekend. > TODO: actually integrate with build system, figure out load paths > etc., also compile files not directly loaded in (lily). Your approach also misses all modules that (lily) pulls in with a (use- modules). Am Montag, dem 21.02.2022 um 22:44 +0100 schrieb Jean Abou Samra: > Honestly, if you are pessimistic about "a month or so" to > get a setup that could work for a stable release, that > doesn't make me confident about making a full switch now ... My statement was about getting proper byte-compilation working, which apparently is now a requirement since this weekend. Before, when I made the proposal, it was only about getting compiled code into the binaries which we already have a solution (GUILE_AUTO_COMPILE) and I just said that, in my opinion, its integration is not a requirement for switching. > As far as I have understood the proposed plan, it involves > putting the master branch in a state that is fundamentally > unsuitable for production in Guile 2 and leave it so > for months, while dropping Guile 1 support, making it > effectively impossible to get something eventually satisfactory > without the success of experimental work that has no > proof-of-concept yet. In an environment run by volunteers > where any person can be "hit by a bus" at any time, I don't > consider this to be a wise plan. I've been working on providing binaries with Guile 2.2 for more than two years now, I had really been looking forward to finally closing this chapter. I was close to giving up a number of times in the past, and I'm considering it again. Realistically speaking, I'm not sure if somebody would be able to continue that journey. Maybe the code I wrote is good enough and I documented enough, but I wouldn't bet on it. Meanwhile, please take a look at reality: Linux distributions are switching to Guile 2.2 for LilyPond, no matter what the state is and what our documentation recommends. Debian attempted to switch for the stable release 2.22.0 (that is now in Debian stable) and we convinced them to stick to Guile 1.8 more-or-less last minute. Arch Linux switched to using Guile 2.2 twice, but I was able to convince them that they should wait. Gentoo has an option to build with Guile 2, not sure if it's used by default or up to the user. The package in Fedora continues to use Guile 2.2 IIRC. What this tells me is that we get a split user community if we do nothing. If we want to avoid that and pro-actively test and fix issues, we have to switch *now* and lead the transition. Jonas signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Hi, Sorry for the late reply, I hoped to have a merge request ready to implement but life is getting in the way. Nevertheless, a proof of concept is this: From 95794324cd4a637c4735447b672a1de91416cc4a Mon Sep 17 00:00:00 2001 From: Jean Abou Samra Date: Sun, 20 Feb 2022 17:00:20 +0100 Subject: [PATCH] WIP: allow separate Guile byte-compilation Benefits: can be integrated into the build system, can byte-compile all files even if not used, can byte-cross-compile. TODO: actually integrate with build system, figure out load paths etc., also compile files not directly loaded in (lily). --- scm/lily.scm | 30 ++ 1 file changed, 30 insertions(+) diff --git a/scm/lily.scm b/scm/lily.scm index afc84d39d8..58d4fc39e4 100644 --- a/scm/lily.scm +++ b/scm/lily.scm @@ -263,6 +263,13 @@ lilypond-book") "Check every property assignment for types.") (clip-systems #f "Generate cut-out snippets of a score.") + (guile-compile #f + "Run Guile byte-compilation on LilyPond's Scheme +files. This is intended for use when compiling or packaging +LilyPond.") + (guile-compile-target #f + "When using -d guile-compile, specifies the +platform triplet to generate bytecode for.") (crop #f "Generate additional, possibly tall single-page output file(s) with cropped margins.") @@ -844,6 +851,11 @@ PIDs or the number of the process." +(use-modules (system base compile) + (system base target)) + +(define start-module (current-module)) + (define-public (lilypond-main files) "Entry point for LilyPond." (eval-string (ly:command-line-code)) @@ -855,6 +867,24 @@ PIDs or the number of the process." (ly:exit 0 #t))) (if (ly:get-option 'gui) (gui-main files)) + (if (ly:get-option 'guile-compile) + (begin + (for-each + (lambda (file) + (let* ((full-path (string-append "lily/" file)) + (file-name (%search-load-path full-path))) + (ly:message "Byte-compiling ~a..." file) + (with-target (let ((custom-target(ly:get-option 'guile-compile-target))) + (if custom-target + (symbol->string custom-target) + %host-type)) + (lambda () + (compile-file file-name + #:env start-module + #:output-file (string-append "/home/jean/repos/lilypond/scm/" file ".go") + ) + init-scheme-files) + (ly:exit 0 #t))) (if (null? files) (begin (ly:usage) (ly:exit 2 #t))) -- 2.32.0 This is in a very experimental stage: it has paths hardcoded for my setup, it doesn't actually load the files, I still have to figure out how to integrate it in Autohell, etc. Nevertheless, it's a proof of concept that it is possible, without restructuring the scm/ directory at all, to get Guile bytecode in a build system for a given architecture. Le 19/02/2022 à 23:21, Jonas Hahnfeld a écrit : Am Samstag, dem 19.02.2022 um 23:05 +0100 schrieb Jean Abou Samra: Le 19/02/2022 à 22:43, Jonas Hahnfeld a écrit : Am Samstag, dem 19.02.2022 um 21:34 +0100 schrieb Jean Abou Samra: Le 19/02/2022 à 21:00, Jonas Hahnfeld a écrit : I'm firmly convinced that the order must be 1. only test with Guile 2.2 in CI 2. make configure only look for Guile 2.2 by default 3. do a release with Guile 2.2 only 4. unless emergency happens, drop all code related to Guile 1.8 very soon after I disagree with 'very soon after' in point 4. Is this code really hurting? We don't have so much of it as far as I can see. All "guile-2" in scm/, GUILEV2 in lily/, and a number of configure checks. It will be significantly easier to restructure all of our Scheme code without that. I am not sure about that. It's unclear to me whether we're going to have to move stuff around in the end -- see the helpful reply from Olivier Dion this evening on guile-user/guile-devel --, The proposed approach is worse than just doing a GUILE_AUTO_COMPILE because we need more code to make it happen. More fundamentally, it doesn't address the cross-compilation setup. It does, see above. and if we do, how would Guile 1 interfere with moving code around? It requires testing, and there are constructs such as eval-when that Guile 1.8 didn't have. Yes, you can work around that, but it will require more code and even more time. (Adressed above.) Plus, your plan of keeping code for Guile 1.8 doesn't work / make sense without keeping GUB working. That is far more complicated and prevents many future changes. Right now, it is in a working state. Because I kept it working so far. I am like St Thomas and need examples to be convinced by 'many'. Even for Cairo there
Re: Blockers for Guile 2.2
Am Samstag, dem 19.02.2022 um 23:21 +0100 schrieb Jonas Hahnfeld via Discussions on LilyPond development: > Am Samstag, dem 19.02.2022 um 23:05 +0100 schrieb Jean Abou Samra: > > > Plus, your plan of keeping code for Guile 1.8 doesn't work / make sense > > > without keeping GUB working. That is far more complicated and prevents > > > many future changes. > > > > Right now, it is in a working state. > > Because I kept it working so far. Oh and btw, it's currently not possible to create a new setup of GUB because the URL for expat 2.1.0 disappeared. It only works if you already have a compiled version of it around. There might be other problems, but I won't bother looking into it. > > I am like St Thomas and need examples to be convinced by 'many'. Even for > > Cairo there is a solution in GUB. > > One example is https://gitlab.com/lilypond/lilypond/-/issues/5831, > another is > https://lists.gnu.org/archive/html/lilypond-devel/2020-08/msg00159.html > where (I think) we shouldn't install the fonts as part of LilyPond's > build system. Both of them would require updates to GUB, which I > honestly don't want to make. https://gitlab.com/lilypond/lilypond/-/merge_requests/1210 will depend on new packages to build the documentation, likely requiring an update of at least the environment GUB can be run in. It's not yet clear (to me) what is needed to make LilyPond fully work with Guile 3.0. If that again involves special code per Guile version, it will be a lot easier if we don't have to support Guile 1.8, 2.2, and version 3.0 at the same time. I potentially would like to rewrite lilypond-invoke-editor in Python because it's the only script using the Guile interpreter. Again, I can't / don't want to do this right now because it would require an update of the wrapper scripts generated by GUB. I'm sure I'll be able to find other issues if you want more. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Am Samstag, dem 19.02.2022 um 23:05 +0100 schrieb Jean Abou Samra: > Le 19/02/2022 à 22:43, Jonas Hahnfeld a écrit : > > Am Samstag, dem 19.02.2022 um 21:34 +0100 schrieb Jean Abou Samra: > > > Le 19/02/2022 à 21:00, Jonas Hahnfeld a écrit : > > > > I'm firmly convinced that the order must be > > > > 1. only test with Guile 2.2 in CI > > > > 2. make configure only look for Guile 2.2 by default > > > > 3. do a release with Guile 2.2 only > > > > 4. unless emergency happens, drop all code related to Guile 1.8 very > > > > soon after > > > I disagree with 'very soon after' in point 4. Is this code > > > really hurting? We don't have so much of it as far as I can > > > see. > > All "guile-2" in scm/, GUILEV2 in lily/, and a number of configure > > checks. It will be significantly easier to restructure all of our > > Scheme code without that. > > I am not sure about that. It's unclear to me whether we're > going to have to move stuff around in the end -- see the helpful > reply from Olivier Dion this evening on guile-user/guile-devel --, The proposed approach is worse than just doing a GUILE_AUTO_COMPILE because we need more code to make it happen. More fundamentally, it doesn't address the cross-compilation setup. > and if we do, how would Guile 1 interfere with moving code > around? It requires testing, and there are constructs such as eval-when that Guile 1.8 didn't have. Yes, you can work around that, but it will require more code and even more time. > > > Plus, your plan of keeping code for Guile 1.8 doesn't work / make sense > > without keeping GUB working. That is far more complicated and prevents > > many future changes. > > Right now, it is in a working state. Because I kept it working so far. > I am like St Thomas and need examples to be convinced by 'many'. Even for > Cairo there is a solution in GUB. One example is https://gitlab.com/lilypond/lilypond/-/issues/5831, another is https://lists.gnu.org/archive/html/lilypond-devel/2020-08/msg00159.html where (I think) we shouldn't install the fonts as part of LilyPond's build system. Both of them would require updates to GUB, which I honestly don't want to make. > To be clear, I'm not happy about keeping it around, but > I very much think we shouldn't actively drop it until we > have a state that resembles what we eventually want to > call stable. Because we don't know what problems are going > to pop up with shipping bytecode. > > > > I find it too risky to lock ourselves into Guile 2 in > > > a state where a stable release would not be satisfactory > > > (see also below). Plus, a number of problems have been popping up > > > with the new binaries. It's great that you're fixing them > > > at the speed of light (thanks!), but I'm not convinced by > > > waiting only shortly after a new release to call it good > > > to go. Would it consume excessive CI minutes if we tested > > > with both Guile 1 and 2 for some time? If so, I would > > > suggest having optional jobs with Guile 1, basically swapping > > > the current roles. > > Again, this doesn't make sense: If we want to keep Guile 1.8 working, > > the jobs must be mandatory. So we're talking at least twice the CI > > minutes, in practice much more because frog can only run one job at a > > time. > > Would that be an acceptable load on the CI infrastructure? It's not only about load, it's about every merge request taking 3x as long to merge. > If not, I wouldn't find too much of an issue with optional jobs. > Right now, we want to keep Guile 2 working, the jobs are > optional, and it keeps working thanks to an occasional > 'hotfix'. Those fixes don't even need to be made while we are > in the state where LilyPond still accepts being configured > with Guile 1. If the transition only takes a month or so, > it should normally remain possible to roll back to Guile 1, > fixing a few things here and there. It is actively dropping > code supporting Guile 1 that I object to at this point. I don't share your optimism here, neither on "occasional 'hotfix'", "a month or so", nor "fixing a few things here and there". I predict things will break, and very quickly if you attempt to do things as invasive as making byte compilation work. Jonas signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Le 19/02/2022 à 22:43, Jonas Hahnfeld a écrit : Am Samstag, dem 19.02.2022 um 21:34 +0100 schrieb Jean Abou Samra: Le 19/02/2022 à 21:00, Jonas Hahnfeld a écrit : I'm firmly convinced that the order must be 1. only test with Guile 2.2 in CI 2. make configure only look for Guile 2.2 by default 3. do a release with Guile 2.2 only 4. unless emergency happens, drop all code related to Guile 1.8 very soon after I disagree with 'very soon after' in point 4. Is this code really hurting? We don't have so much of it as far as I can see. All "guile-2" in scm/, GUILEV2 in lily/, and a number of configure checks. It will be significantly easier to restructure all of our Scheme code without that. I am not sure about that. It's unclear to me whether we're going to have to move stuff around in the end -- see the helpful reply from Olivier Dion this evening on guile-user/guile-devel --, and if we do, how would Guile 1 interfere with moving code around? Plus, your plan of keeping code for Guile 1.8 doesn't work / make sense without keeping GUB working. That is far more complicated and prevents many future changes. Right now, it is in a working state. I am like St Thomas and need examples to be convinced by 'many'. Even for Cairo there is a solution in GUB. To be clear, I'm not happy about keeping it around, but I very much think we shouldn't actively drop it until we have a state that resembles what we eventually want to call stable. Because we don't know what problems are going to pop up with shipping bytecode. I find it too risky to lock ourselves into Guile 2 in a state where a stable release would not be satisfactory (see also below). Plus, a number of problems have been popping up with the new binaries. It's great that you're fixing them at the speed of light (thanks!), but I'm not convinced by waiting only shortly after a new release to call it good to go. Would it consume excessive CI minutes if we tested with both Guile 1 and 2 for some time? If so, I would suggest having optional jobs with Guile 1, basically swapping the current roles. Again, this doesn't make sense: If we want to keep Guile 1.8 working, the jobs must be mandatory. So we're talking at least twice the CI minutes, in practice much more because frog can only run one job at a time. Would that be an acceptable load on the CI infrastructure? If not, I wouldn't find too much of an issue with optional jobs. Right now, we want to keep Guile 2 working, the jobs are optional, and it keeps working thanks to an occasional 'hotfix'. Those fixes don't even need to be made while we are in the state where LilyPond still accepts being configured with Guile 1. If the transition only takes a month or so, it should normally remain possible to roll back to Guile 1, fixing a few things here and there. It is actively dropping code supporting Guile 1 that I object to at this point. Jean
Re: Blockers for Guile 2.2
Am Samstag, dem 19.02.2022 um 21:34 +0100 schrieb Jean Abou Samra: > Le 19/02/2022 à 21:00, Jonas Hahnfeld a écrit : > > I'm firmly convinced that the order must be > > 1. only test with Guile 2.2 in CI > > 2. make configure only look for Guile 2.2 by default > > 3. do a release with Guile 2.2 only > > 4. unless emergency happens, drop all code related to Guile 1.8 very > > soon after > > I disagree with 'very soon after' in point 4. Is this code > really hurting? We don't have so much of it as far as I can > see. All "guile-2" in scm/, GUILEV2 in lily/, and a number of configure checks. It will be significantly easier to restructure all of our Scheme code without that. Plus, your plan of keeping code for Guile 1.8 doesn't work / make sense without keeping GUB working. That is far more complicated and prevents many future changes. > I find it too risky to lock ourselves into Guile 2 in > a state where a stable release would not be satisfactory > (see also below). Plus, a number of problems have been popping up > with the new binaries. It's great that you're fixing them > at the speed of light (thanks!), but I'm not convinced by > waiting only shortly after a new release to call it good > to go. Would it consume excessive CI minutes if we tested > with both Guile 1 and 2 for some time? If so, I would > suggest having optional jobs with Guile 1, basically swapping > the current roles. Again, this doesn't make sense: If we want to keep Guile 1.8 working, the jobs must be mandatory. So we're talking at least twice the CI minutes, in practice much more because frog can only run one job at a time. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Am Samstag, dem 19.02.2022 um 21:23 +0100 schrieb David Kastrup: > Jonas Hahnfeld via Discussions on LilyPond development > writes: > > > Our answers are racing here, I'd suggest we keep that discussion to the > > sub-thread of your original reply. Just for completeness and future > > reference: Setting GUILE_AUTO_COMPILE=1 is an explicit choice by users. > > You don't get this by default, hence no problems. > > If you are prepared to call "severe performance degradation of default > installations" no problem. For the next few development releases, yes. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Le 19/02/2022 à 21:00, Jonas Hahnfeld a écrit : Am Samstag, dem 19.02.2022 um 19:58 +0100 schrieb Jean Abou Samra: Le 19/02/2022 à 17:57, Jonas Hahnfeld a écrit : Hi all, I'd like to discuss what are considered blocker issues for a switch to Guile 2.2. After the release of 2.23.6, there were reports of major problems on Windows, namely that the binaries were broken when extracted with the Windows Explorer (#6281) and that file names with special characters didn't work (#6282). I think I found solutions for both of them, either already merged (!1194 for #6281) or in review (!1219 for #6282). The second large topic was performance of the binaries with Guile 2.2, which we know will be worse without compiled bytecode. In https://lists.gnu.org/archive/html/lilypond-devel/2022-02/msg00099.html Jean writes [Guile bytecode for LilyPond's .scm files] should be added eventually before we make a full switch. I don't fully agree and think that bytecode compilation shouldn't block the switch. In my opinion, it would be fine for the next development releases to be somewhat slower if that results in Guile 2.2 being available sooner. The trade-off will be different for a stable release where we certainly want LilyPond to be as fast as possible. That was the main reason why I posted about GUILE_AUTO_COMPILE last year, to have a fallback ready in case we can't get proper compilation working (and showing that the compiled bytecode gives very similar performance to Guile 1.8, of course). I am not sure we are talking about the same issue. Here I was talking about shipping .go files with the binaries. I wonder if you interpreted it as getting compiled code with 'guild compile' rather than GUILE_AUTO_COMPILE. No, I didn't. GUILE_AUTO_COMPILE is the only way of compiling that we have right now. I think it's OK if the bytecode files are generated by GUILE_AUTO_COMPILE (I have been looking into making 'guild compile' work but don't have meaningful results to report yet). I don't think we must have bytecode files in the installers in order to retire GUB and start making releases exclusively with the new system. I do think we should ship bytecode in the installers before we make a stable release with Guile 2, or before we completely drop support for Guile 1 in the sense of removing it from CI and starting to accept code that doesn't work with it. I find these statements contradicting. As soon was we start doing releases with Guile 2.2, this is the only version that we must test and that we must develop for. It doesn't make sense to have developers working with Guile 1.8 locally if users are on Guile 2.2. I'm firmly convinced that the order must be 1. only test with Guile 2.2 in CI 2. make configure only look for Guile 2.2 by default 3. do a release with Guile 2.2 only 4. unless emergency happens, drop all code related to Guile 1.8 very soon after I disagree with 'very soon after' in point 4. Is this code really hurting? We don't have so much of it as far as I can see. I find it too risky to lock ourselves into Guile 2 in a state where a stable release would not be satisfactory (see also below). Plus, a number of problems have been popping up with the new binaries. It's great that you're fixing them at the speed of light (thanks!), but I'm not convinced by waiting only shortly after a new release to call it good to go. Would it consume excessive CI minutes if we tested with both Guile 1 and 2 for some time? If so, I would suggest having optional jobs with Guile 1, basically swapping the current roles. In my opinion, Guile 1 can be dropped completely once we have had a few months of a released LilyPond version that works the way we want for everyone -- including acceptable solutions for downstream distributors. Yes, this does mean still more time before we have forgotten about Guile 1, but I think it would be a mistake to rush this delicate transition. You need to get them; ie the scripts need to compile files such that all .scm files are compiled (including files like accreg.scm that need to be loaded explicitly), and then collect all of them and "install" them into the right place. For cross-compilation to mingw, you need to collect them from the native Linux binaries. How hard is that? To have all files compiled, you could run the regression test suite for example. Our answers are racing here, I'd suggest we keep that discussion to the sub-thread of your original reply. Just for completeness and future reference: Setting GUILE_AUTO_COMPILE=1 is an explicit choice by users. You don't get this by default, hence no problems. OK, that's the part I hadn't understood. But then, no, I don't think this makes for acceptable versions of LilyPond. The startup time without bytecode is darn annoying. So, yes, in my opinion, having bytecode files preinstalled in binaries is a blocker for eliminating Guile 1 (but not for making development releases with Guile 2 only as long as Guile 1 support is not removed).
Re: Blockers for Guile 2.2
Jonas Hahnfeld via Discussions on LilyPond development writes: > Our answers are racing here, I'd suggest we keep that discussion to the > sub-thread of your original reply. Just for completeness and future > reference: Setting GUILE_AUTO_COMPILE=1 is an explicit choice by users. > You don't get this by default, hence no problems. If you are prepared to call "severe performance degradation of default installations" no problem. -- David Kastrup
Re: Blockers for Guile 2.2
Jonas Hahnfeld writes: > Am Samstag, dem 19.02.2022 um 21:08 +0100 schrieb David Kastrup: >> Jonas Hahnfeld writes: >> >> > Am Samstag, dem 19.02.2022 um 18:14 +0100 schrieb David Kastrup: >> > > That is not as much a speed issue as an stability issue. The byte >> > > compilation caches are not robust across updates and downgrades since >> > > they are based on file names. >> > >> > ... where the path includes the version of LilyPond. Additionally, >> > Guile also checks the modification date. >> >> And the modification date is the date of unpacking on all platforms? > > For Unix, it's the timestamp when the binaries were built. For Windows, > it depends. What I was trying to say here is that a user modifying a > source .scm file will not silently get wrongly compiled byte-code. scm files will not change as much changed by the user rather than by installation and/or a distribution installer. Those don't use version dependent directories. I think that without a working distribution strategy, it would be a mistake to release LilyPond and let packagers figure out on their own how they are going to deal with the byte compilation mess. -- David Kastrup
Re: Blockers for Guile 2.2
Am Samstag, dem 19.02.2022 um 21:08 +0100 schrieb David Kastrup: > Jonas Hahnfeld writes: > > > Am Samstag, dem 19.02.2022 um 18:14 +0100 schrieb David Kastrup: > > > That is not as much a speed issue as an stability issue. The byte > > > compilation caches are not robust across updates and downgrades since > > > they are based on file names. > > > > ... where the path includes the version of LilyPond. Additionally, > > Guile also checks the modification date. > > And the modification date is the date of unpacking on all platforms? For Unix, it's the timestamp when the binaries were built. For Windows, it depends. What I was trying to say here is that a user modifying a source .scm file will not silently get wrongly compiled byte-code. > > > > If our own scm files are not installing > > > with their individual set of .go files, the installations bleed over > > > into the user domain and remnants may cause problems. > > > > I'm not sure I understand your concern here. For LilyPond, compiled > > files never end up in the user's $HOME directory but always within > > versioned directories within share/ (with GUILE_AUTO_COMPILE=1). > > Why would the user have write permissions there? What happens with > other users also running LilyPond? Our answers are racing here, I'd suggest we keep that discussion to the sub-thread of your original reply. Just for completeness and future reference: Setting GUILE_AUTO_COMPILE=1 is an explicit choice by users. You don't get this by default, hence no problems. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Jonas Hahnfeld writes: > Am Samstag, dem 19.02.2022 um 18:14 +0100 schrieb David Kastrup: >> That is not as much a speed issue as an stability issue. The byte >> compilation caches are not robust across updates and downgrades since >> they are based on file names. > > ... where the path includes the version of LilyPond. Additionally, > Guile also checks the modification date. And the modification date is the date of unpacking on all platforms? >> If our own scm files are not installing >> with their individual set of .go files, the installations bleed over >> into the user domain and remnants may cause problems. > > I'm not sure I understand your concern here. For LilyPond, compiled > files never end up in the user's $HOME directory but always within > versioned directories within share/ (with GUILE_AUTO_COMPILE=1). Why would the user have write permissions there? What happens with other users also running LilyPond? -- David Kastrup
Re: Blockers for Guile 2.2
Am Samstag, dem 19.02.2022 um 21:04 +0100 schrieb David Kastrup: > Jonas Hahnfeld via Discussions on LilyPond development > writes: > > > > Having byte-compiled files in the > > > user's cache removes the ability to completely uninstall and negates > > > the ease of moving the installation around with static binaries. > > > > Byte-compiled files from LilyPond never end up in the user's cache, > > they always go to the versioned directory in share/ (as you write in > > your other reply). > > How would that work without the user having write permission there? It doesn't. Compilation is an explicit choice by users right now, by setting an environment variable. If they do this and can't write in the installation directory, their problem. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Am Samstag, dem 19.02.2022 um 18:14 +0100 schrieb David Kastrup: > Jonas Hahnfeld via Discussions on LilyPond development > writes: > > > Hi all, > > > > I'd like to discuss what are considered blocker issues for a switch to > > Guile 2.2. > > > > After the release of 2.23.6, there were reports of major problems on > > Windows, namely that the binaries were broken when extracted with the > > Windows Explorer (#6281) and that file names with special characters > > didn't work (#6282). I think I found solutions for both of them, either > > already merged (!1194 for #6281) or in review (!1219 for #6282). > > > > The second large topic was performance of the binaries with Guile 2.2, > > which we know will be worse without compiled bytecode. In > > https://lists.gnu.org/archive/html/lilypond-devel/2022-02/msg00099.html > > Jean writes > > > [Guile bytecode for LilyPond's .scm files] should be added eventually > > before we make a full switch. > > > > I don't fully agree and think that bytecode compilation shouldn't block > > the switch. In my opinion, it would be fine for the next development > > releases to be somewhat slower if that results in Guile 2.2 being > > available sooner. > > That is not as much a speed issue as an stability issue. The byte > compilation caches are not robust across updates and downgrades since > they are based on file names. ... where the path includes the version of LilyPond. Additionally, Guile also checks the modification date. > If our own scm files are not installing > with their individual set of .go files, the installations bleed over > into the user domain and remnants may cause problems. I'm not sure I understand your concern here. For LilyPond, compiled files never end up in the user's $HOME directory but always within versioned directories within share/ (with GUILE_AUTO_COMPILE=1). Also the user must be explicit about getting them, in which case they are on their own. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Jonas Hahnfeld via Discussions on LilyPond development writes: >> Having byte-compiled files in the >> user's cache removes the ability to completely uninstall and negates >> the ease of moving the installation around with static binaries. > > Byte-compiled files from LilyPond never end up in the user's cache, > they always go to the versioned directory in share/ (as you write in > your other reply). How would that work without the user having write permission there? -- David Kastrup
Re: Blockers for Guile 2.2
Am Samstag, dem 19.02.2022 um 19:58 +0100 schrieb Jean Abou Samra: > Le 19/02/2022 à 17:57, Jonas Hahnfeld a écrit : > > Hi all, > > > > I'd like to discuss what are considered blocker issues for a switch to > > Guile 2.2. > > > > After the release of 2.23.6, there were reports of major problems on > > Windows, namely that the binaries were broken when extracted with the > > Windows Explorer (#6281) and that file names with special characters > > didn't work (#6282). I think I found solutions for both of them, either > > already merged (!1194 for #6281) or in review (!1219 for #6282). > > > > The second large topic was performance of the binaries with Guile 2.2, > > which we know will be worse without compiled bytecode. In > > https://lists.gnu.org/archive/html/lilypond-devel/2022-02/msg00099.html > > Jean writes > > > [Guile bytecode for LilyPond's .scm files] should be added eventually > > before we make a full switch. > > > > I don't fully agree and think that bytecode compilation shouldn't block > > the switch. In my opinion, it would be fine for the next development > > releases to be somewhat slower if that results in Guile 2.2 being > > available sooner. The trade-off will be different for a stable release > > where we certainly want LilyPond to be as fast as possible. That was > > the main reason why I posted about GUILE_AUTO_COMPILE last year, to > > have a fallback ready in case we can't get proper compilation working > > (and showing that the compiled bytecode gives very similar performance > > to Guile 1.8, of course). > > > > I am not sure we are talking about the same issue. Here I was talking > about shipping .go files with the binaries. I wonder if you > interpreted it as getting compiled code with 'guild compile' rather > than GUILE_AUTO_COMPILE. No, I didn't. GUILE_AUTO_COMPILE is the only way of compiling that we have right now. > I think it's OK if the bytecode files are > generated by GUILE_AUTO_COMPILE (I have been looking into making > 'guild compile' work but don't have meaningful results to report > yet). I don't think we must have bytecode files in the installers in > order to retire GUB and start making releases exclusively with the new > system. I do think we should ship bytecode in the installers before we > make a stable release with Guile 2, or before we completely drop > support for Guile 1 in the sense of removing it from CI and starting > to accept code that doesn't work with it. I find these statements contradicting. As soon was we start doing releases with Guile 2.2, this is the only version that we must test and that we must develop for. It doesn't make sense to have developers working with Guile 1.8 locally if users are on Guile 2.2. I'm firmly convinced that the order must be 1. only test with Guile 2.2 in CI 2. make configure only look for Guile 2.2 by default 3. do a release with Guile 2.2 only 4. unless emergency happens, drop all code related to Guile 1.8 very soon after > Having byte-compiled files in the > user's cache removes the ability to completely uninstall and negates > the ease of moving the installation around with static binaries. Byte-compiled files from LilyPond never end up in the user's cache, they always go to the versioned directory in share/ (as you write in your other reply). > Plus, > the logging output with all warnings and ';;;' can look scaring for > Joe User -- I worry that less computer-educated people are going > to think LilyPond is doing something bad to their system. Only if users set GUILE_AUTO_COMPILE=1 themselves, and there are far worse environment variables to break about every program on your computer. FWIW think it's bad to tell users about this variable (but that harm is already done). > There > is also a big issue with the way Guile determines if a bytecode > file is up-to-date. I could be wrong, but as far as I can see, > it takes any bytecode that has a newer date than the source, which > means that if we produce bytecode via GUILE_AUTO_COMPILE in the > user's environment, and the user tries to downgrade LilyPond to an > older version while keeping the install in the same location, it > will break mysteriously because the cached .go file will still have > a newer timestamp than the corresponding .scm file, and thus won't be > recompiled. (already addressed by your reply) > Could you elaborate on the technical difficulties posed by > shipping .go files? You need to get them; ie the scripts need to compile files such that all .scm files are compiled (including files like accreg.scm that need to be loaded explicitly), and then collect all of them and "install" them into the right place. For cross-compilation to mingw, you need to collect them from the native Linux binaries. signature.asc Description: This is a digitally signed message part
Re: Blockers for Guile 2.2
Le 19/02/2022 à 19:58, Jean Abou Samra a écrit : There is also a big issue with the way Guile determines if a bytecode file is up-to-date. I could be wrong, but as far as I can see, it takes any bytecode that has a newer date than the source, which means that if we produce bytecode via GUILE_AUTO_COMPILE in the user's environment, and the user tries to downgrade LilyPond to an older version while keeping the install in the same location, it will break mysteriously because the cached .go file will still have a newer timestamp than the corresponding .scm file, and thus won't be recompiled. Ah, but I see now that there is the version number in the path after share/. So this particular point is moot, sorry.
Re: Blockers for Guile 2.2
Le 19/02/2022 à 17:57, Jonas Hahnfeld a écrit : Hi all, I'd like to discuss what are considered blocker issues for a switch to Guile 2.2. After the release of 2.23.6, there were reports of major problems on Windows, namely that the binaries were broken when extracted with the Windows Explorer (#6281) and that file names with special characters didn't work (#6282). I think I found solutions for both of them, either already merged (!1194 for #6281) or in review (!1219 for #6282). The second large topic was performance of the binaries with Guile 2.2, which we know will be worse without compiled bytecode. In https://lists.gnu.org/archive/html/lilypond-devel/2022-02/msg00099.html Jean writes [Guile bytecode for LilyPond's .scm files] should be added eventually before we make a full switch. I don't fully agree and think that bytecode compilation shouldn't block the switch. In my opinion, it would be fine for the next development releases to be somewhat slower if that results in Guile 2.2 being available sooner. The trade-off will be different for a stable release where we certainly want LilyPond to be as fast as possible. That was the main reason why I posted about GUILE_AUTO_COMPILE last year, to have a fallback ready in case we can't get proper compilation working (and showing that the compiled bytecode gives very similar performance to Guile 1.8, of course). I am not sure we are talking about the same issue. Here I was talking about shipping .go files with the binaries. I wonder if you interpreted it as getting compiled code with 'guild compile' rather than GUILE_AUTO_COMPILE. I think it's OK if the bytecode files are generated by GUILE_AUTO_COMPILE (I have been looking into making 'guild compile' work but don't have meaningful results to report yet). I don't think we must have bytecode files in the installers in order to retire GUB and start making releases exclusively with the new system. I do think we should ship bytecode in the installers before we make a stable release with Guile 2, or before we completely drop support for Guile 1 in the sense of removing it from CI and starting to accept code that doesn't work with it. Having byte-compiled files in the user's cache removes the ability to completely uninstall and negates the ease of moving the installation around with static binaries. Plus, the logging output with all warnings and ';;;' can look scaring for Joe User -- I worry that less computer-educated people are going to think LilyPond is doing something bad to their system. There is also a big issue with the way Guile determines if a bytecode file is up-to-date. I could be wrong, but as far as I can see, it takes any bytecode that has a newer date than the source, which means that if we produce bytecode via GUILE_AUTO_COMPILE in the user's environment, and the user tries to downgrade LilyPond to an older version while keeping the install in the same location, it will break mysteriously because the cached .go file will still have a newer timestamp than the corresponding .scm file, and thus won't be recompiled. Could you elaborate on the technical difficulties posed by shipping .go files? What do you think? Are there are other issues that need to be taken into account? Performance *with* bytecode. I am thinking to the messages from Thomas Scharkowski where he reports LilyPond being twice as slow as normal even with it. (I will soon write another email to ask him to take some debugging steps so we can hopefully understand if something in particular is causing this slowdown, like Fontconfig regenerating its cache every time or such). Thanks, Jean
Re: Blockers for Guile 2.2
Jonas Hahnfeld via Discussions on LilyPond development writes: > Hi all, > > I'd like to discuss what are considered blocker issues for a switch to > Guile 2.2. > > After the release of 2.23.6, there were reports of major problems on > Windows, namely that the binaries were broken when extracted with the > Windows Explorer (#6281) and that file names with special characters > didn't work (#6282). I think I found solutions for both of them, either > already merged (!1194 for #6281) or in review (!1219 for #6282). > > The second large topic was performance of the binaries with Guile 2.2, > which we know will be worse without compiled bytecode. In > https://lists.gnu.org/archive/html/lilypond-devel/2022-02/msg00099.html > Jean writes >> [Guile bytecode for LilyPond's .scm files] should be added eventually > before we make a full switch. > > I don't fully agree and think that bytecode compilation shouldn't block > the switch. In my opinion, it would be fine for the next development > releases to be somewhat slower if that results in Guile 2.2 being > available sooner. That is not as much a speed issue as an stability issue. The byte compilation caches are not robust across updates and downgrades since they are based on file names. If our own scm files are not installing with their individual set of .go files, the installations bleed over into the user domain and remnants may cause problems. -- David Kastrup
Blockers for Guile 2.2
Hi all, I'd like to discuss what are considered blocker issues for a switch to Guile 2.2. After the release of 2.23.6, there were reports of major problems on Windows, namely that the binaries were broken when extracted with the Windows Explorer (#6281) and that file names with special characters didn't work (#6282). I think I found solutions for both of them, either already merged (!1194 for #6281) or in review (!1219 for #6282). The second large topic was performance of the binaries with Guile 2.2, which we know will be worse without compiled bytecode. In https://lists.gnu.org/archive/html/lilypond-devel/2022-02/msg00099.html Jean writes > [Guile bytecode for LilyPond's .scm files] should be added eventually before we make a full switch. I don't fully agree and think that bytecode compilation shouldn't block the switch. In my opinion, it would be fine for the next development releases to be somewhat slower if that results in Guile 2.2 being available sooner. The trade-off will be different for a stable release where we certainly want LilyPond to be as fast as possible. That was the main reason why I posted about GUILE_AUTO_COMPILE last year, to have a fallback ready in case we can't get proper compilation working (and showing that the compiled bytecode gives very similar performance to Guile 1.8, of course). What do you think? Are there are other issues that need to be taken into account? Jonas signature.asc Description: This is a digitally signed message part
