Re: State of LilyPond with Guile 2.2
On 4/12/2021 7:06 AM, David Kastrup wrote: Not being able to use 64-bit addressing on Windows with GUILE 1.8 is an extremely serious problem. I was of the opinion that we distributed a 64-bit version here? Or did I get that wrong? Windows Subsystem for Linux continues to gain capability. It's allowing for Linux executables called from Windows and vice versa. I understand the Windows-native LilyPond build consumes the most maintenance resources of all the builds. Maybe eventually it's easier to have a PowerShell script or something that sets up everything LilyPond needs under the Windows Subsystem for Linux? Downside: needs a more-recent Windows 10 to do good with this. -- Karlin High Missouri, USA
Re: State of LilyPond with Guile 2.2
Am Montag, dem 12.04.2021 um 13:50 +0200 schrieb David Kastrup: > Werner LEMBERG writes: > > > > > ... you were talking about advancing to Guile 2.x as the next step, > > > > and if I have understood your original e-mail correctly, this speed > > > > is only available with 3.x. > > > > > > Then please re-read the initial message *carefully*! > > > > I stand corrected. Somehow I got the impression that compiled Scheme > > code is a 3.x thing only. Obviously I'm getting old. > > Guile-1.8 already compiles Scheme code. Guile-2.0 has a virtual machine > and does byte compilation for that into separate files. Guile-2.2 has a > different virtual machine. I've not kept track of what happened since > then. There may have been experiments with JIT compilation and/or > machine code but I don't really know. Yes, Guile 3.0 has a JIT (on x86 at least), see https://www.gnu.org/software/guile/news/gnu-guile-300-released.html signature.asc Description: This is a digitally signed message part
Re: State of LilyPond with Guile 2.2
Han-Wen Nienhuys writes: > It seems that most of the core work on GUILE is done by a single > person (Andy Wingo). Last time I looked at commit history and mailing list usage, the "development version" of Guile was exclusively Andy Wingo's domain and he did not communicate with anyone on publicly available channels about what he was doing, and the changes were partly of the back-and-forth kind for experiments that may or may not fail. Anyone else was constrained to "stable" branch work. Which very much consisted in bugfixes, but also comparatively well-defined additions and changes that would properly belong in a development branch except that there is no such thing outside of Andy Wingo's domain. Bug fixes in the stable branch might or might not make it into the development branch. There was some cohesion of the Andy and non-Andy work into the 3.0 release (which was after some of the few non-trivial other developers like Mark Weavers left the project in frustration). I have not followed development since then and don't know what happened afterwards. > If he leaves the project for one reason or another, is there anyone > who can productively get things done on the GUILE codebase? I don't > think I am confident about becoming versed in Scheme enough to tinker > with an optimizing compiler written in a dynamically typed language. If there is a compiler bug or design problem to get fixed, I guess that's curtains or workaround time. Outside of the compiler, there have been significant fixes, but the number of people willing to invest work here has shrunk. > Not being able to use 64-bit addressing on Windows with GUILE 1.8 is > an extremely serious problem. I was of the opinion that we distributed a 64-bit version here? Or did I get that wrong? -- David Kastrup
Re: State of LilyPond with Guile 2.2
Werner LEMBERG writes: >>> ... you were talking about advancing to Guile 2.x as the next step, >>> and if I have understood your original e-mail correctly, this speed >>> is only available with 3.x. >> >> Then please re-read the initial message *carefully*! > > I stand corrected. Somehow I got the impression that compiled Scheme > code is a 3.x thing only. Obviously I'm getting old. Guile-1.8 already compiles Scheme code. Guile-2.0 has a virtual machine and does byte compilation for that into separate files. Guile-2.2 has a different virtual machine. I've not kept track of what happened since then. There may have been experiments with JIT compilation and/or machine code but I don't really know. -- David Kastrup
Re: State of LilyPond with Guile 2.2
Am Montag, dem 12.04.2021 um 10:00 +0200 schrieb Han-Wen Nienhuys: > On Mon, Apr 12, 2021 at 9:36 AM Jonas Hahnfeld wrote: > > > > Am Montag, dem 12.04.2021 um 09:28 +0200 schrieb Han-Wen Nienhuys: > > > Not being able to use 64-bit addressing on Windows with GUILE 1.8 is > > > an extremely serious problem. What is the reason for this? Is it > > > because dynamic loading doesn't work correctly, and GUILE tries to > > > load SRFI modules as .dlls ? > > > > No, the dynamic loading works on Windows (after all, how would 32 bit > > work otherwise?) The issue is that 64 bit on Windows uses a very weird > > ABI and long's are 4 bytes whereas pointers are 8 bytes. Guile isn't > > prepared to handle this, and the integrated GC fails in wonderful ways > > on this issue. If you want to know more, there's quite a bit of > > material just one search away. > > I did a search, but clearly not the right one :) - do you have some pointers? https://github.com/slowphil/mingw-w64-guile1.8/blob/master/X86_64_building.md is pretty easy to find. I don't have more links right now, but I seem to remember some post on guile-devel from some years ago? My commit logs for the work-in-progress binary build says that I eventually succeeded in compiling, but then experienced division by zero errors, if that information helps... Jonas signature.asc Description: This is a digitally signed message part
Re: State of LilyPond with Guile 2.2
On Mon, Apr 12, 2021 at 9:36 AM Jonas Hahnfeld wrote: > > Am Montag, dem 12.04.2021 um 09:28 +0200 schrieb Han-Wen Nienhuys: > > Not being able to use 64-bit addressing on Windows with GUILE 1.8 is > > an extremely serious problem. What is the reason for this? Is it > > because dynamic loading doesn't work correctly, and GUILE tries to > > load SRFI modules as .dlls ? > > No, the dynamic loading works on Windows (after all, how would 32 bit > work otherwise?) The issue is that 64 bit on Windows uses a very weird > ABI and long's are 4 bytes whereas pointers are 8 bytes. Guile isn't > prepared to handle this, and the integrated GC fails in wonderful ways > on this issue. If you want to know more, there's quite a bit of > material just one search away. I did a search, but clearly not the right one :) - do you have some pointers? > Not sure if I mentioned this last year, > but I actually tried to fix in Guile and gave up after a week or so. I > seriously don't think that fixing this ourselves when a solution via > bdwgc is readily available is a good investment of resources. I had a look at GUILE's git history, and unfortunately, the BDGCW change is one that was entangled with lots of other changes in the run-up to 2.0. Sigh. I am quite familiar with the 1.8 garbage collector, so I'm curious where the fundamental problems come from, but given the traction of the fixes I sent to TTN, investigating this for real would only be an option if we went with the 1.8 fork route. -- Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
Re: State of LilyPond with Guile 2.2
>> ... you were talking about advancing to Guile 2.x as the next step, >> and if I have understood your original e-mail correctly, this speed >> is only available with 3.x. > > Then please re-read the initial message *carefully*! I stand corrected. Somehow I got the impression that compiled Scheme code is a 3.x thing only. Obviously I'm getting old. Werner
Re: State of LilyPond with Guile 2.2
Am Montag, dem 12.04.2021 um 09:28 +0200 schrieb Han-Wen Nienhuys: > Not being able to use 64-bit addressing on Windows with GUILE 1.8 is > an extremely serious problem. What is the reason for this? Is it > because dynamic loading doesn't work correctly, and GUILE tries to > load SRFI modules as .dlls ? No, the dynamic loading works on Windows (after all, how would 32 bit work otherwise?) The issue is that 64 bit on Windows uses a very weird ABI and long's are 4 bytes whereas pointers are 8 bytes. Guile isn't prepared to handle this, and the integrated GC fails in wonderful ways on this issue. If you want to know more, there's quite a bit of material just one search away. Not sure if I mentioned this last year, but I actually tried to fix in Guile and gave up after a week or so. I seriously don't think that fixing this ourselves when a solution via bdwgc is readily available is a good investment of resources. Jonas signature.asc Description: This is a digitally signed message part
Re: State of LilyPond with Guile 2.2
Am Montag, dem 12.04.2021 um 09:19 +0200 schrieb Werner LEMBERG: > > > Note that I don't want that we stay with Guile 1.8 forever, but > > > the slowness of 2.x and 3.x is a serious issue. To sacrifice > > > this still enormous speed advantage just for the sake of > > > orthodoxy seems wrong to me. > > > > So 10% performance regression for users is enormous? > > This would be acceptable, but... > > > After all, this was the *real* purpose of this thread, which nobody > > bothered answering so far. > > ... you were talking about advancing to Guile 2.x as the next step, > and if I have understood your original e-mail correctly, this speed > is only available with 3.x. Then please re-read the initial message *carefully*! signature.asc Description: This is a digitally signed message part
Re: State of LilyPond with Guile 2.2
On Sun, Apr 11, 2021 at 7:55 PM Werner LEMBERG wrote: > > Guile 2.2 also makes binary distribution much nicer (because there > > no more shared srfi libraries, so lilypond can be linked as one > > static executable) and makes it possible to offer 64 bit executables > > for Windows. > > ... especially to support some 64bit architectures. > > > But given the reactions, I'll reduce activity on my work towards > > Guile 2.2... > > Hopefully there is not a misunderstanding: I very much appreciate your > activities (and I'm quite sure that Han-Wen thinks the same) and > invaluable tests! > > It is very unfortunate that more recent Guile versions cause such a > serious deterioration for LilyPond. Maybe it makes sense for the > foreseeable future to stay with the status quo, this is, using Guile > 1.8 as much as possible, and offering support for 2.x (or 3.x) for > platforms where version 1.8 can't be used. And if we go this route it > makes sense to do what Han-Wen suggests, namely to add Guile 1.8 as a > submodule. Jonas didn't include 3.0 in the benchmarking because it's not generally available yet, but I think it is relevant data. Part of the reason why we want to be on an up-to-date release of GUILE is the promise that future releases will be better for LilyPond. By looking at 3.0, we can predict what the user experience will be in a few years. I am mainly worried that GUILE will further evolve into an optimizing compiler system, where the compilation takes more time, the interpreted code runs more slowly, and the build process creates increasingly complicated requirements. In this sense, the move to GUILE 2+ is a watershed moment: once we drop support for 1.8 (dropping GC glue), it will be really hard to go back, and we basically sign up to follow GUILE in their roadmap. By contrast, GUILE 1.8 is mostly C, so there is a fair chance that we can fix problems we find by ourselves. It seems that most of the core work on GUILE is done by a single person (Andy Wingo). If he leaves the project for one reason or another, is there anyone who can productively get things done on the GUILE codebase? I don't think I am confident about becoming versed in Scheme enough to tinker with an optimizing compiler written in a dynamically typed language. Not being able to use 64-bit addressing on Windows with GUILE 1.8 is an extremely serious problem. What is the reason for this? Is it because dynamic loading doesn't work correctly, and GUILE tries to load SRFI modules as .dlls ?
Re: State of LilyPond with Guile 2.2
>> Note that I don't want that we stay with Guile 1.8 forever, but the >> slowness of 2.x and 3.x is a serious issue. To sacrifice this >> still enormous speed advantage just for the sake of orthodoxy seems >> wrong to me. > > So 10% performance regression for users is enormous? This would be acceptable, but... > After all, this was the *real* purpose of this thread, which nobody > bothered answering so far. ... you were talking about advancing to Guile 2.x as the next step, and if I have understood your original e-mail correctly, this speed is only available with 3.x. Werner
Re: State of LilyPond with Guile 2.2
Am Montag, dem 12.04.2021 um 09:00 +0200 schrieb Werner LEMBERG: > > > Almost all developers use a Unix-like OS and can be thus served > > > with Guile 1.8.x! Are there actually LilyPond developers who > > > work > > > natively with Windows or MacOS? With 'natively' I mean using a > > > binary specifically compiled for that platform and not a virtual > > > box. > > > > Adding a mix of Guile versions will make the situation much worse > > because we know for sure that 1.8 and 2.2 are sufficiently > > different > > that it can cause bugs on their own. > > While this is true, I believe... > > > Claiming support for both will make reproducing issues much harder, > > and things will get outright horrible if, say, we continue to offer > > 32 bit for Windows using Guile 1.8 and 64 bit with Guile 2.2. > > ... that this is greatly exaggerated. AFAICS, we have managed that > quite well up to now (partially due to your heroic efforts). Yes, because nobody uses it. Just look at the number of GC related crashes that I fixed in the past months, and I know of at least two more that I didn't get the chance to analyze in full detail yet. > > > Also consider what this means for extension writers: They'll have > > to take two Guile versions into account and possibly test both of > > them. I fear this split will be equally bad as Python 2 vs 3 was > > over years... > > Are the differences for users really that significant? Looking into > LilyPond's `scm` directory I only see very low-level stuff that needs > explicit use of `guile-2`. Maybe not as bad as Python 2 vs 3, but there are differences and heavy use of Scheme will break. > Note that I don't want that we stay with Guile 1.8 forever, but the > slowness of 2.x and 3.x is a serious issue. To sacrifice this still > enormous speed advantage just for the sake of orthodoxy seems wrong > to me. So 10% performance regression for users is enormous? What would be acceptable, in your opinion? After all, this was the *real* purpose of this thread, which nobody bothered answering so far. Jonas signature.asc Description: This is a digitally signed message part
Re: State of LilyPond with Guile 2.2
>> Almost all developers use a Unix-like OS and can be thus served >> with Guile 1.8.x! Are there actually LilyPond developers who work >> natively with Windows or MacOS? With 'natively' I mean using a >> binary specifically compiled for that platform and not a virtual >> box. > > Adding a mix of Guile versions will make the situation much worse > because we know for sure that 1.8 and 2.2 are sufficiently different > that it can cause bugs on their own. While this is true, I believe... > Claiming support for both will make reproducing issues much harder, > and things will get outright horrible if, say, we continue to offer > 32 bit for Windows using Guile 1.8 and 64 bit with Guile 2.2. ... that this is greatly exaggerated. AFAICS, we have managed that quite well up to now (partially due to your heroic efforts). > Also consider what this means for extension writers: They'll have to > take two Guile versions into account and possibly test both of them. > I fear this split will be equally bad as Python 2 vs 3 was over > years... Are the differences for users really that significant? Looking into LilyPond's `scm` directory I only see very low-level stuff that needs explicit use of `guile-2`. Note that I don't want that we stay with Guile 1.8 forever, but the slowness of 2.x and 3.x is a serious issue. To sacrifice this still enormous speed advantage just for the sake of orthodoxy seems wrong to me. Werner
Re: State of LilyPond with Guile 2.2
Am Sonntag, dem 11.04.2021 um 22:19 +0200 schrieb Werner LEMBERG: > > > It is very unfortunate that more recent Guile versions cause such > > > a serious deterioration for LilyPond. Maybe it makes sense for > > > the foreseeable future to stay with the status quo, this is, > > > using Guile 1.8 as much as possible, > > > > For reference: Ubuntu 16.04, the longest supported version to have > > a package for Guile 1.8, will go EOL on April 30; Debian Jessie > > already is dead since June 2020. > > Well, Guile 1.8.x served us surprisingly well in the last years. It > still has a maintainer, and I'm not aware of serious bugs that are > showstoppers for LilyPond. > > > > and offering support for 2.x (or 3.x) for platforms where version > > > 1.8 can't be used. > > > > The problem I see is that whatever is part of the official binaries > > will get the most attention, or we risk giving users a > > configuration that developers don't test regularly. Which means > > the other version simply won't matter and carry on their own bugs. > > But we already have this situation: Almost all developers use a > Unix-like OS and can be thus served with Guile 1.8.x! Are there > actually LilyPond developers who work natively with Windows or MacOS? > With 'natively' I mean using a binary specifically compiled for that > platform and not a virtual box. Adding a mix of Guile versions will make the situation much worse because we know for sure that 1.8 and 2.2 are sufficiently different that it can cause bugs on their own. Claiming support for both will make reproducing issues much harder, and things will get outright horrible if, say, we continue to offer 32 bit for Windows using Guile 1.8 and 64 bit with Guile 2.2. Also consider what this means for extension writers: They'll have to take two Guile versions into account and possibly test both of them. I fear this split will be equally bad as Python 2 vs 3 was over years... Jonas signature.asc Description: This is a digitally signed message part
