Re: GUILE 2/3 and string encoding cost

[Han-Wen Nienhuys]

On Wed, Jan 22, 2020 at 11:43 PM David Kastrup  wrote:

> > Actually, the I was comparing the -O2 build with the -O0 build.
> >
> > When recompiling, the Scheme init (reading .scm files) takes 0.31s in 1.8
> > vs. 2.7s in 2.0, a 9x slowdown.
>
> The Guile-2 compiler is doing a lot of optimisations, and LilyPond's
> startup code switches off byte compilation because the dependencies are
> hard to get under control.


where does this happen?

The current codebase at least manages to
> avoid to compile code with as-yet undefined macros, something that
> Guile-1.8 had no problems with but Guile-2.0 refuses.
>

Do you mean that we don't have them anymore, or is there something else
going on?


>
> So the Scheme loading speed is sort-of expected due to Guile relying on
> byte compilation for speed and we switch it off.
>
>
Much to the contrary. Byte-compiling is slow (but running it should be
faster), something you can see from building guile. If it is switched off,
we are getting the "fast" experience.

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwe
?

Re: packaging lilypond as a docker container?

[Jan Nieuwenhuizen]

David Kastrup writes:

> I did not remember you posting this at the time, and my reply at that
> time
> 
> appears enfuriatingly unrelated.  Basically I don't comment at all but
> rather complain about GUB.

Ah.

> As previously said: I don't worry much about installers for the free
> UNIX-like systems since people prefer going by distribution installers
> on those in general.  What we have to cover definitely is Windows, and
> preferably MacOSX (but the latter one appears to have sailed, at least
> in connection with GUB, unless we find some OpenDarwin-like approach for
> 64bit).

Ok.  You probably know much better what best helps most users.

> I have no experience with Docker containers, but the posting from 2016
> you referenced appears to rather point to a native binary?

Oops...I tried to reference the mingw lilypond prototype but the URL
I sent pointed to this post

https://lists.gnu.org/archive/html/guix-devel/2016-03/msg01116.html

which showed

$ ./pre-inst-env guix environment --ad-hoc wine -- wine 
/gnu/store/2j8qaqbc9q1w3z6k054nmp7j432yxfq7-hello-2.10/bin/hello.exe
Hello, world!

the first mingw32 success.  Oh well.

janneke

-- 
Jan Nieuwenhuizen  | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: packaging lilypond as a docker container?

[Jan Nieuwenhuizen]

Karlin High writes:

> Am I understanding it correctly, that the Guix project in the linked
> message successfully used MinGW to make a "Hello, World" command line
> program for Windows?

Yes.  I finished the mingw cross build of guile-2.x so we have that in
Guix now.  Currently I am using that with Mike Gran's yet unmerged
wip-mingw-guile-2.2 branch to build Guile binaries for Windows.

At the time I rewrote the native lilypond build, I think some
configure/autoconf patches were needed for Guix) and prototyped the
mingw cross build

https://gitlab.com/janneke/guix/tree/wip-mingw32+lilypond

that would need some work to finish and merge.

Since then, Carl Dong has implemented the bitcoin build and mingw cross
build; as a divergence here is a nice talk about why he did that

https://www.youtube.com/watch?v=I2iShmUTEl8

He has stated to have interest in developing a MacOS cross build.

Greetings,
janneke

-- 
Jan Nieuwenhuizen  | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: packaging lilypond as a docker container?

[Karlin High]

On 1/22/2020 7:23 AM, Jan Nieuwenhuizen wrote:

FWIW, four years ago I created a prototype

 https://lists.gnu.org/archive/html/lilypond-devel/2016-03/msg00204.html


Thanks a lot for the reference, copied to notes for responses to future 
mailing list GUB questions.


Am I understanding it correctly, that the Guix project in the linked 
message successfully used MinGW to make a "Hello, World" command line 
program for Windows?

--
Karlin High
Missouri, USA

Re: GUILE 2/3 and string encoding cost

[Thomas Morley]

Am Mi., 22. Jan. 2020 um 22:32 Uhr schrieb Thomas Morley
:
>
> Am Mi., 22. Jan. 2020 um 12:12 Uhr schrieb Thomas Morley
> :
> >
> > Am Mi., 22. Jan. 2020 um 12:02 Uhr schrieb David Kastrup :
> > >
> > > Han-Wen Nienhuys  writes:
> >
> > > > So, what hard data do we have on GUILE 2/3 slowness, and what does
> > > > that data say?
> > >
> > > That data says "humongous slowdown".  There is not much more than
> > > speculation what this is caused by as far as I know.
> >
> > I can't provide any insight here.
> > Though, once I have a working LilyPond/guile-3, I'll test how
> > "humongous" it will.
> >
> > A working LilyPond/guile-3 means: successfull make, make doc, make
> > test-baseline.
> > Currently only the first is done.
>
> I've now got a successful ´make LANGS='' doc´
>
> ´make test-baseline´ still fails, as it did with all recent guile-versions.
> Though, I don't understand why it does so. Compiling the docs will
> return the regression-tests as html/pdf already.
> So all files are already successfully compiled.
> What's the difference?
>
> Next I'll apply Han-Wen's patch about ly_scm_write_string
> https://sourceforge.net/p/testlilyissues/issues/5666/
> (Earlier I suspected ly_scm_write_string to be the problem)
>
>
> Cheers,
>   Harm

Regrettable applying the patch for Issue 5666 doesn't solve the
problem, ´make test-baseline´ still errors, without any useful
message.

Cheers,
  Harm

Re: GUILE 2/3 and string encoding cost

[David Kastrup]

Han-Wen Nienhuys  writes:

> On Wed, Jan 22, 2020 at 10:53 PM Han-Wen Nienhuys  wrote:
>
>>
>>
>> On Wed, Jan 22, 2020 at 12:01 PM David Kastrup  wrote:
>>
>>>
>>> > So, what hard data do we have on GUILE 2/3 slowness, and what does
>>> > that data say?
>>>
>>> That data says "humongous slowdown".  There is not much more than
>>> speculation what this is caused by as far as I know.
>>>
>>
>> I can see the 2x slowdown, and it looks uniformly distributed over the
>> whole process. The GUILE 2.0 release
>>
>
> Actually, the I was comparing the -O2 build with the -O0 build.
>
> When recompiling, the Scheme init (reading .scm files) takes 0.31s in 1.8
> vs. 2.7s in 2.0, a 9x slowdown.

The Guile-2 compiler is doing a lot of optimisations, and LilyPond's
startup code switches off byte compilation because the dependencies are
hard to get under control.  The current codebase at least manages to
avoid to compile code with as-yet undefined macros, something that
Guile-1.8 had no problems with but Guile-2.0 refuses.

So the Scheme loading speed is sort-of expected due to Guile relying on
byte compilation for speed and we switch it off.

-- 
David Kastrup

Re: Looking for help in configuring LilyDev in VirtualBox

[Michael Käppler]

Am 22.01.2020 um 14:03 schrieb Peter Toye:

Re: Looking for help in configuring LilyDev in VirtualBox
> Yes, I would suggest that you start from scratch again.

I made a new guest machine using your instructions. There are a few
issues here:

Step 3: the file names are LilyDev-1-debian-vm and the raw fle is
.iso not .raw

Thanks, I fixed this.


Step11: I got an error from the command:
[dev@LilyDev:~]$ sudo dpkg-reconfigure keyboard-layout
[sudo] password for dev:
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog
based frontend cannot be used. at
/usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline
dpkg-query: package 'keyboard-layout' is not installed and no
information is available
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.
/usr/sbin/dpkg-reconfigure: keyboard-layout is not installed
[dev@LilyDev:~]$

So I used the mechanism in the current CG Step 8 to change the
keyboard - probably a better idea and more user-friendly. Unless, of
course, the clipboard is working.

That's because the keyboard-layout package is not installed in LilyDev
v1 by default. In LilyDev v2, it will be there.


After all that I tried your idea of starting the clipboard manually -
it worked!! Or I'd never have managed to get the above text right :)

So the basic question is: why isn't it starting up? I used to be a
Solaris sysadmin, but that ended 20 years ago and my copy of Frisch is
long gone.

I don't really know because I can't reproduce this behaviour on my machine.

Best regards,
Michael



> Best regards,
> Michael



Best regards,

Peter

Re: GUILE 2/3 and string encoding cost

[Han-Wen Nienhuys]

On Wed, Jan 22, 2020 at 10:53 PM Han-Wen Nienhuys  wrote:

>
>
> On Wed, Jan 22, 2020 at 12:01 PM David Kastrup  wrote:
>
>>
>> > So, what hard data do we have on GUILE 2/3 slowness, and what does
>> > that data say?
>>
>> That data says "humongous slowdown".  There is not much more than
>> speculation what this is caused by as far as I know.
>>
>
> I can see the 2x slowdown, and it looks uniformly distributed over the
> whole process. The GUILE 2.0 release
>

Actually, the I was comparing the -O2 build with the -O0 build.

When recompiling, the Scheme init (reading .scm files) takes 0.31s in 1.8
vs. 2.7s in 2.0, a 9x slowdown.





>   https://lwn.net/Articles/428288/
>
> has one big red flag for me.
>
>   * Switch to the Boehm-Demers-Weiser garbage collector
>
>   Guile now uses the Boehm-Demers-Weiser conservative garbage collector
>   (aka. libgc).  It makes interaction with C code easier making, for
>   instance, the use of mark and free SMOB procedures optional in many
>   cases.  It also improves performance.
>
> let me get out the profiler to see what is going on.
>
> --
> Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen
>


-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: GUILE 2/3 and string encoding cost

[Han-Wen Nienhuys]

On Wed, Jan 22, 2020 at 12:01 PM David Kastrup  wrote:

>
> > So, what hard data do we have on GUILE 2/3 slowness, and what does
> > that data say?
>
> That data says "humongous slowdown".  There is not much more than
> speculation what this is caused by as far as I know.
>

I can see the 2x slowdown, and it looks uniformly distributed over the
whole process. The GUILE 2.0 release

  https://lwn.net/Articles/428288/

has one big red flag for me.

  * Switch to the Boehm-Demers-Weiser garbage collector

  Guile now uses the Boehm-Demers-Weiser conservative garbage collector
  (aka. libgc).  It makes interaction with C code easier making, for
  instance, the use of mark and free SMOB procedures optional in many
  cases.  It also improves performance.

let me get out the profiler to see what is going on.

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: GUILE 2/3 and string encoding cost

[Thomas Morley]

Am Mi., 22. Jan. 2020 um 22:36 Uhr schrieb Thomas Morley
:
>
> Am Mi., 22. Jan. 2020 um 21:52 Uhr schrieb Karlin High :
> >
> > On 1/22/2020 2:07 PM, Han-Wen Nienhuys wrote:
> > > Do we have a standardized test file for benchmarking performance?
> >
> > I can't speak to "standardized," but I do remember some threads that had
> > benchmarking going on by various users, using large LilyPond projects.
> >
> > The Robert Carver "Missa Dum Sacrum Mysterium" from Vaughan McAlley.
> > (114 pages)
> >
> > 
>
> I took that file and compiled it with 2.19.83 and LilyPond-2.21.0/guile-3

(NB: without patch for Issue 5666 applied)

> Although, I didn't apply convert-ly, which may be a problem or not.
>
> The results:
>
> 2.19.83
>
> 1. run
>
> real3m38,352s
> user3m24,493s
> sys0m4,659s
>
> 2. run
>
> real3m33,635s
> user3m23,291s
> sys0m4,563s
>
>
> 2.21.0 with guile-3
>
> 1.run
>
> real9m41,223s
> user11m36,337s
> sys0m3,600s
>
> 2. run
>
> real9m25,902s
> user11m20,798s
> sys0m3,743s
>
> Cheers,
>   Harm

Re: GUILE 2/3 and string encoding cost

[Thomas Morley]

Am Mi., 22. Jan. 2020 um 21:52 Uhr schrieb Karlin High :
>
> On 1/22/2020 2:07 PM, Han-Wen Nienhuys wrote:
> > Do we have a standardized test file for benchmarking performance?
>
> I can't speak to "standardized," but I do remember some threads that had
> benchmarking going on by various users, using large LilyPond projects.
>
> The Robert Carver "Missa Dum Sacrum Mysterium" from Vaughan McAlley.
> (114 pages)
>
> 

I took that file and compiled it with 2.19.83 and LilyPond-2.21.0/guile-3
Although, I didn't apply convert-ly, which may be a problem or not.

The results:

2.19.83

1. run

real3m38,352s
user3m24,493s
sys0m4,659s

2. run

real3m33,635s
user3m23,291s
sys0m4,563s


2.21.0 with guile-3

1.run

real9m41,223s
user11m36,337s
sys0m3,600s

2. run

real9m25,902s
user11m20,798s
sys0m3,743s

Cheers,
  Harm

Re: GUILE 2/3 and string encoding cost

[Thomas Morley]

Am Mi., 22. Jan. 2020 um 12:12 Uhr schrieb Thomas Morley
:
>
> Am Mi., 22. Jan. 2020 um 12:02 Uhr schrieb David Kastrup :
> >
> > Han-Wen Nienhuys  writes:
>
> > > So, what hard data do we have on GUILE 2/3 slowness, and what does
> > > that data say?
> >
> > That data says "humongous slowdown".  There is not much more than
> > speculation what this is caused by as far as I know.
>
> I can't provide any insight here.
> Though, once I have a working LilyPond/guile-3, I'll test how
> "humongous" it will.
>
> A working LilyPond/guile-3 means: successfull make, make doc, make
> test-baseline.
> Currently only the first is done.

I've now got a successful ´make LANGS='' doc´

´make test-baseline´ still fails, as it did with all recent guile-versions.
Though, I don't understand why it does so. Compiling the docs will
return the regression-tests as html/pdf already.
So all files are already successfully compiled.
What's the difference?

Next I'll apply Han-Wen's patch about ly_scm_write_string
https://sourceforge.net/p/testlilyissues/issues/5666/
(Earlier I suspected ly_scm_write_string to be the problem)


Cheers,
  Harm

Re: GUILE 2/3 and string encoding cost

[Karlin High]

On 1/22/2020 2:07 PM, Han-Wen Nienhuys wrote:

Do we have a standardized test file for benchmarking performance?


I can't speak to "standardized," but I do remember some threads that had 
benchmarking going on by various users, using large LilyPond projects.


The Robert Carver "Missa Dum Sacrum Mysterium" from Vaughan McAlley. 
(114 pages)




And the Heinrich Schütz "Schwanengesang" Psalm 119 from Brent Annable. 
(194 pages)



--
Karlin High
Missouri, USA

Re: GUILE 2/3 and string encoding cost

[Urs Liska]

Am Mittwoch, den 22.01.2020, 20:28 + schrieb Carl Sorensen:
> 
> On 1/22/20, 1:21 PM, "lilypond-devel on behalf of David Kastrup" <
> lilypond-devel-bounces+c_sorensen=byu@gnu.org on behalf of 
> d...@gnu.org> wrote:
> 
> Han-Wen Nienhuys  writes:
> 
> > On Wed, Jan 22, 2020 at 12:01 PM David Kastrup 
> wrote:
> >
> >> Han-Wen Nienhuys  writes:
> >>
> >> > I looked a bit through the GUILE source code to see what is
> going on.
> >> >
> >> > I believe our current hypothesis (LilyPond's slowdown is
> caused by
> >> > expensive unicode transcoding into 32-bit strings) is
> incorrect.
> >> >
> >> > If you look into the source code, you can see that the UTF-8 
> -> SCM
> >> > conversion checks if there are any code points over 255
> >> >
> >> >
> >> >
> >> 
> https://git.savannah.nongnu.org/cgit/guile.git//tree/libguile/strings.c/?id=1b8e9ca0e37fab366435436995248abdfc780a10#n1620
> >> >
> >> > if there aren't, it uses Latin1 encoding ("narrow == 1") to
> encode the
> >> > string as a normal byte array. This code walks the string
> twice, but that
> >> > is very cheap due to CPU cache locality, so it should be
> >> > essentially equivalent to whatever GUILE 1.8 was doing.
> >>
> >> GUILE 1.8 did not walk the string even once
> >>
> >
> > GUILE 1.8 walks it once when you do memcpy.
> 
> Ok, but that's sort of a cheap walk.
> 
> >> > Even so, if the input flie does use UTF-8, there should be
> little
> >> > overhead, because the number of texts that we process is
> always
> >> > small. LilyPond is not a text processor.
> >> >
> >> > So, what hard data do we have on GUILE 2/3 slowness, and
> what does
> >> > that data say?
> >>
> >> That data says "humongous slowdown".  There is not much more
> than
> >> speculation what this is caused by as far as I know.
> >>
> >>
> > Do we have a standardized test file for benchmarking
> performance?
> 
> input/regression/mozart-hrn-3.ly possibly, but it's not
> particularly
> large.
> 
> We don't have a standardized test file, but we do have some
> representative results from a couple of (unknown but described)
> files:
> 
> https://lists.gnu.org/archive/html/lilypond-devel/2018-10/msg00054.html
> 
> Perhaps we could get those files to become standards (along with some
> other, shorter-compiling files).
> 

Not right now but in the not-so-distant future I'd be able¹ to provide
the 650 examples from the Mozart violin school as a set of many small
scores, which might be a nice complement to one large score.

Urs

¹ It's not about copyright (the edition is released under a CC) but
about being ready for that purpose. 

> Carl
>  
> 

Re: GUILE 2/3 and string encoding cost

[Carl Sorensen]

On 1/22/20, 1:21 PM, "lilypond-devel on behalf of David Kastrup" 
 
wrote:

Han-Wen Nienhuys  writes:

> On Wed, Jan 22, 2020 at 12:01 PM David Kastrup  wrote:
>
>> Han-Wen Nienhuys  writes:
>>
>> > I looked a bit through the GUILE source code to see what is going on.
>> >
>> > I believe our current hypothesis (LilyPond's slowdown is caused by
>> > expensive unicode transcoding into 32-bit strings) is incorrect.
>> >
>> > If you look into the source code, you can see that the UTF-8 -> SCM
>> > conversion checks if there are any code points over 255
>> >
>> >
>> >
>> 
https://git.savannah.nongnu.org/cgit/guile.git//tree/libguile/strings.c/?id=1b8e9ca0e37fab366435436995248abdfc780a10#n1620
>> >
>> > if there aren't, it uses Latin1 encoding ("narrow == 1") to encode the
>> > string as a normal byte array. This code walks the string twice, but 
that
>> > is very cheap due to CPU cache locality, so it should be
>> > essentially equivalent to whatever GUILE 1.8 was doing.
>>
>> GUILE 1.8 did not walk the string even once
>>
>
> GUILE 1.8 walks it once when you do memcpy.

Ok, but that's sort of a cheap walk.

>> > Even so, if the input flie does use UTF-8, there should be little
>> > overhead, because the number of texts that we process is always
>> > small. LilyPond is not a text processor.
>> >
>> > So, what hard data do we have on GUILE 2/3 slowness, and what does
>> > that data say?
>>
>> That data says "humongous slowdown".  There is not much more than
>> speculation what this is caused by as far as I know.
>>
>>
> Do we have a standardized test file for benchmarking performance?

input/regression/mozart-hrn-3.ly possibly, but it's not particularly
large.

We don't have a standardized test file, but we do have some representative 
results from a couple of (unknown but described) files:

https://lists.gnu.org/archive/html/lilypond-devel/2018-10/msg00054.html

Perhaps we could get those files to become standards (along with some other, 
shorter-compiling files).

Carl
 

Re: GUILE 2/3 and string encoding cost

[David Kastrup]

Han-Wen Nienhuys  writes:

> On Wed, Jan 22, 2020 at 12:01 PM David Kastrup  wrote:
>
>> Han-Wen Nienhuys  writes:
>>
>> > I looked a bit through the GUILE source code to see what is going on.
>> >
>> > I believe our current hypothesis (LilyPond's slowdown is caused by
>> > expensive unicode transcoding into 32-bit strings) is incorrect.
>> >
>> > If you look into the source code, you can see that the UTF-8 -> SCM
>> > conversion checks if there are any code points over 255
>> >
>> >
>> >
>> https://git.savannah.nongnu.org/cgit/guile.git//tree/libguile/strings.c/?id=1b8e9ca0e37fab366435436995248abdfc780a10#n1620
>> >
>> > if there aren't, it uses Latin1 encoding ("narrow == 1") to encode the
>> > string as a normal byte array. This code walks the string twice, but that
>> > is very cheap due to CPU cache locality, so it should be
>> > essentially equivalent to whatever GUILE 1.8 was doing.
>>
>> GUILE 1.8 did not walk the string even once
>>
>
> GUILE 1.8 walks it once when you do memcpy.

Ok, but that's sort of a cheap walk.

>> > Even so, if the input flie does use UTF-8, there should be little
>> > overhead, because the number of texts that we process is always
>> > small. LilyPond is not a text processor.
>> >
>> > So, what hard data do we have on GUILE 2/3 slowness, and what does
>> > that data say?
>>
>> That data says "humongous slowdown".  There is not much more than
>> speculation what this is caused by as far as I know.
>>
>>
> Do we have a standardized test file for benchmarking performance?

input/regression/mozart-hrn-3.ly possibly, but it's not particularly
large.

-- 
David Kastrup

Re: GUILE 2/3 and string encoding cost

[Han-Wen Nienhuys]

On Wed, Jan 22, 2020 at 12:01 PM David Kastrup  wrote:

> Han-Wen Nienhuys  writes:
>
> > I looked a bit through the GUILE source code to see what is going on.
> >
> > I believe our current hypothesis (LilyPond's slowdown is caused by
> > expensive unicode transcoding into 32-bit strings) is incorrect.
> >
> > If you look into the source code, you can see that the UTF-8 -> SCM
> > conversion checks if there are any code points over 255
> >
> >
> >
> https://git.savannah.nongnu.org/cgit/guile.git//tree/libguile/strings.c/?id=1b8e9ca0e37fab366435436995248abdfc780a10#n1620
> >
> > if there aren't, it uses Latin1 encoding ("narrow == 1") to encode the
> > string as a normal byte array. This code walks the string twice, but that
> > is very cheap due to CPU cache locality, so it should be
> > essentially equivalent to whatever GUILE 1.8 was doing.
>
> GUILE 1.8 did not walk the string even once
>

GUILE 1.8 walks it once when you do memcpy.


> > Even so, if the input flie does use UTF-8, there should be little
> > overhead, because the number of texts that we process is always
> > small. LilyPond is not a text processor.
> >
> > So, what hard data do we have on GUILE 2/3 slowness, and what does
> > that data say?
>
> That data says "humongous slowdown".  There is not much more than
> speculation what this is caused by as far as I know.
>
>
Do we have a standardized test file for benchmarking performance?

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: Set page breaking properties in the System grob (issue 581470047 by hanw...@gmail.com)

[dak]

On 2020/01/22 18:51:07, Erlend Aasland wrote:
> On 2020/01/22 16:20:21, Dan Eble wrote:
> > Veto! I've had more than one bad experience with lowercase ell.
> > 
> > In one case--I think it was elsewhere in LilyPond--it was purely a
readability
> > issue, but reason enough to avoid it.
> > 
> > In the other, one person *thought* that he was using it as a loop
variable (i,
> > j, k, l, ...), though he was actually using it outside its loop. 
His code
> > compiled anyway because another person had defined a global of the
same name. 
> > Thankfully, I don't work at that company anymore.
> > 
> > I suggest "SCM line = lines;" as an alternative.
> 
> +1

Interesting.  I'd have no qualms about l as a local loop variable,
though who uses this as a global variable is definitely asking for a
code review.  I consider it likely that GCC would deliver a warning for
the described situation, but it is of course valid code.

https://codereview.appspot.com/581470047/

Re: Set page breaking properties in the System grob (issue 581470047 by hanw...@gmail.com)

[erlenda]

On 2020/01/22 16:20:21, Dan Eble wrote:
> Veto! I've had more than one bad experience with lowercase ell.
> 
> In one case--I think it was elsewhere in LilyPond--it was purely a
readability
> issue, but reason enough to avoid it.
> 
> In the other, one person *thought* that he was using it as a loop
variable (i,
> j, k, l, ...), though he was actually using it outside its loop.  His
code
> compiled anyway because another person had defined a global of the
same name. 
> Thankfully, I don't work at that company anymore.
> 
> I suggest "SCM line = lines;" as an alternative.

+1

https://codereview.appspot.com/581470047/

Re: packaging lilypond as a docker container?

[Dan Eble]

On Jan 21, 2020, at 11:01, Michael Käppler  wrote:
> 
> Would be happy to try the Dockerfile, since I've never used Docker
> before and find
> the concept interesting.
> Could you push your changes to your LilyDev fork and/or file a pull
> request against
> Federico's repo?

Federico has merged a couple changes already.  This is the last one I have 
planned:
https://github.com/fedelibre/LilyDev/pull/19
— 
Dan

Set page breaking properties in the System grob (issue 581470047 by hanw...@gmail.com)

[nine . fierce . ballads]

What's the purpose of this change?  Is it preparation for a new feature?


https://codereview.appspot.com/581470047/diff/565490043/lily/page-breaking.cc
File lily/page-breaking.cc (right):

https://codereview.appspot.com/581470047/diff/565490043/lily/page-breaking.cc#newcode624
lily/page-breaking.cc:624: for (SCM l = lines; scm_is_pair (l); l =
scm_cdr (l))
Veto! I've had more than one bad experience with lowercase ell.

In one case--I think it was elsewhere in LilyPond--it was purely a
readability issue, but reason enough to avoid it.

In the other, one person *thought* that he was using it as a loop
variable (i, j, k, l, ...), though he was actually using it outside its
loop.  His code compiled anyway because another person had defined a
global of the same name.  Thankfully, I don't work at that company
anymore.

I suggest "SCM line = lines;" as an alternative.

https://codereview.appspot.com/581470047/

Issue #4550 Avoid "using" directives in included files

[Dan Eble]

Begin forwarded message:
> 
> From: "pkx166h" 
> Subject: [testlilyissues:issues] #4550 Avoid "using" directives in included 
> files
> Date: January 22, 2020 at 10:13:39 EST
> To: "[testlilyissues:issues] " 
> <4...@issues.testlilyissues.p.re.sourceforge.net>
> Reply-To: "[testlilyissues:issues] " 
> <4...@issues.testlilyissues.p.re.sourceforge.net>
> 
> Patch: new --> review
> Comment:
> Passes make, make check and a full make doc.
> 
If there is someone here willing to spend the personal and machine time to test 
this patch in GUB, it would be very helpful.  The first attempts at this change 
(some years ago) were plagued by issues that didn't show up until after the 
changes were pushed.  I expect that they have been fixed, but it would be nice 
to be sure before pushing.

https://codereview.appspot.com/579240043/
— 
Dan

Re: Context paths (and the Edition Engraver)

[Kieren MacMillan]

Hi Jan-Peter (et al.),

> I am amazed what kind of discussion is raised on this topic :)

Me, too. Maybe we should have international music engraving conferences more 
often than once every 20 years…  ;)

> I'd suggest alternative commands to create something
> like an XQuery/CSS/whatever functionality.

+1

> To have the possibility to address contexts *like* in CSS has some
> appeal. But IMHO it shouldn't disturb the current input scheme.

+1

> If the ID is a string (or symbol) the path can be easily constructed
> with ly:context-parent. Something like:

Nice!
(Aside: Now that I’ve gotten past my Schemenangst, I look at that code and 
immediately see how it works and its potential uses and extensions.)

Cheers,
Kieren.


Kieren MacMillan, composer (he/him/his)
‣ website: www.kierenmacmillan.info
‣ email: i...@kierenmacmillan.info

Re: packaging lilypond as a docker container?

[Karlin High]

On 1/22/2020 4:36 AM, David Kastrup wrote:

I have no experience with Docker and containers.


The one time I tried Docker on Windows, I was put off by the setup 
effort that seemed intended to initiate the user into an entire 
worldview and ecosystem. Docker seemingly didn't provide an easy way to 
download their software without creating an account at their place 
first. That's behavior I expect from Google, Apple, and Microsoft, not 
an open-source project. After some digging, I think there was a direct 
download link somewhere, given in a forum post. But then after the 
install, Docker seemed pretty non-functional until I logged into my 
local software with a Docker account.


Okay, okay... I sort-of get why they're doing this, I can deal with it.

BUT THEN the command-line window for the Docker container kept going 
non-responsive on me, needing a kill with Windows Task Manager. Research 
turned up the sorts of sprawling forum threads that I'm used to seeing 
for failed Windows updates. Lots of suggestions, most aren't helping, 
the problem seemed like an unsolved mystery. And the answer that seemed 
most promising would have been non-trivial to implement.


At that point, I retreated back to VirtualBox, Hyper-V, and Windows 
Subsystem for Linux. I was getting too far removed from LilyPond to not 
feel like I was pursuing a dead end.


IN DEFENSE OF DOCKER, all of that is from memory and I can't recover the 
exact details without research. I'm sure my experience is not 
representative and that Docker is actually great. No way could it have 
as large of a following as it does otherwise. I'm just unfortunate to 
have never seen Docker's best stuff or completely understand why I need 
it. Someday™ I plan to try again.


But the conclusion I came to is that setup effort for Docker on Windows 
is not much different than for VirtualBox. (Unless, of course, someone's 
going for the full VirtualBox experience with host-guest shared 
clipboard like in that other thread right now. I never got that working 
on LilyDev, either, making do with network file shares and such to move 
things to and from the VM.)


Docker on Windows is apparently changing a lot, too. Windows Subsystem 
for Linux is now going for a full Linux kernel, and allowing Linux file 
access and execution from the Windows environment containing it. Haven't 
checked for a while, not sure where Microsoft is at with that effort, 
but Docker seems to be building on it.



--
Karlin High
Missouri, USA

Re: packaging lilypond as a docker container?

[David Kastrup]

Jan Nieuwenhuizen  writes:

> David Kastrup writes:
>
>> Han-Wen Nienhuys  writes:
>>> GUIX is Jan's current project. I think it has some similarities to
>>> GUB, but it is focused on providing an environment where all binaries
>>> are reproducibly built from source. This is much more ambitious than
>>> GUB, and seems overkill compared to what we need for LilyPond. I think
>>> using it also entails many more compilation steps, which would makes
>>> the release process slow again.
>>
>> I don't think it has an answer for the elephant in the room: Windows.
>
> FWIW, four years ago I created a prototype
>
> https://lists.gnu.org/archive/html/lilypond-devel/2016-03/msg00204.html
>
> enough for me to decide never to touch GUB again.  Don't get me wrong,
> doing GUB was great and I could make some good use of what we learned to
> redo this in Guix; I just think creating a custom build for LilyPond is
> not making good use of our time.
>
> Creating a LilyPond [cross] build that nobody uses is also not very
> inspiring; so I abandoned/paused the project at the time.

I did not remember you posting this at the time, and my reply at that
time

appears enfuriatingly unrelated.  Basically I don't comment at all but
rather complain about GUB.

Trying to beat sense into what I wrote at the time, this was an
endorsement for replacing GUB with Guix, given that it appears to cover
the respective bases.

> You are probably aware that Guix produces generic, relocatable
> GNU/Linux binaries as well as docker containers.

As previously said: I don't worry much about installers for the free
UNIX-like systems since people prefer going by distribution installers
on those in general.  What we have to cover definitely is Windows, and
preferably MacOSX (but the latter one appears to have sailed, at least
in connection with GUB, unless we find some OpenDarwin-like approach for
64bit).

I have no experience with Docker containers, but the posting from 2016
you referenced appears to rather point to a native binary?

-- 
David Kastrup

Re: packaging lilypond as a docker container?

[Jan Nieuwenhuizen]

David Kastrup writes:

> Han-Wen Nienhuys  writes:
>> GUIX is Jan's current project. I think it has some similarities to
>> GUB, but it is focused on providing an environment where all binaries
>> are reproducibly built from source. This is much more ambitious than
>> GUB, and seems overkill compared to what we need for LilyPond. I think
>> using it also entails many more compilation steps, which would makes
>> the release process slow again.
>
> I don't think it has an answer for the elephant in the room: Windows.

FWIW, four years ago I created a prototype

https://lists.gnu.org/archive/html/lilypond-devel/2016-03/msg00204.html

enough for me to decide never to touch GUB again.  Don't get me wrong,
doing GUB was great and I could make some good use of what we learned to
redo this in Guix; I just think creating a custom build for LilyPond is
not making good use of our time.

Creating a LilyPond [cross] build that nobody uses is also not very
inspiring; so I abandoned/paused the project at the time.

You are probably aware that Guix produces generic, relocatable GNU/Linux
binaries as well as docker containers.

Greetings,
janneke

-- 
Jan Nieuwenhuizen  | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Re: patches without tracker issues

[James]

Hello
---
Regards

James



On 22.01.2020 11:06, Han-Wen Nienhuys wrote:
these were sent before I had access, I think. Can you folksk have  a 
look?


https://codereview.appspot.com/559370043/

This is https://sourceforge.net/p/testlilyissues/issues/5669/



https://codereview.appspot.com/581470047/

This is https://sourceforge.net/p/testlilyissues/issues/5677/

regards

James

Re: Looking for help in configuring LilyDev in VirtualBox

[Peter Toye]

Wednesday, January 22, 2020, 6:43:48 AM, Michael Käppler wrote:


> Am 21.01.2020 um 20:44 schrieb Peter Toye:
>> Re: Looking for help in configuring LilyDev in VirtualBox
>> This time after the reboot I got the same result. No sharing of
>> clipboard, no more processes. Tried closing VBox and restarting it,
>> but same result.
> Try to start the client manually and see what happens:
> /usr/bin/VBoxClient --clipboard

>>
>> There's one slight oddity: often (but not always) when booting the
>> LilyDev guest I get an error message - a bit too long to copy the text
>> so here's a screenshot:
>>
>> I don't know if it's significant. It doesn't change the clipboard
>> behaviour.
>  From time to time, this happens on my machine, too.
> I thought it would be a problem specific to my
> combination of VirtualBox
> version / host os
> and guest os, but if you encounter the same problem it could be a
> general issue.

It's not important as long as it's not significant. I thought it might be 
affecting the boot process.

>> It's possible that my hamfisted attempts to install LilyDev as a guest
>> without all the instructions has screwed something up. I could start
>> again tomorrow using the instructions you sent me and see if it makes
>> any difference.
> Yes, I would suggest that you start from scratch again.

I made a new guest machine using your instructions. There are a few issues 
here: 
 
Step 3: the file names are LilyDev-1-debian-vm and the raw fle is .iso not .raw

Step11: I got an error from the command:
[dev@LilyDev:~]$ sudo dpkg-reconfigure keyboard-layout
[sudo] password for dev: 
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog based 
frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 
76.)
debconf: falling back to frontend: Readline
dpkg-query: package 'keyboard-layout' is not installed and no information is 
available
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.
/usr/sbin/dpkg-reconfigure: keyboard-layout is not installed
[dev@LilyDev:~]$

So I used the mechanism in the current CG Step 8 to change the keyboard - 
probably a better idea and more user-friendly. Unless, of course, the clipboard 
is working.

After all that I tried your idea of starting the clipboard manually - it 
worked!! Or I'd never have managed to get the above text right :)

So the basic question is: why isn't it starting up? I used to be a Solaris 
sysadmin, but that ended 20 years ago and my copy of Frisch is long gone.

> Best regards,
> Michael



Best regards,

Peter

   

Re: Re: packaging lilypond as a docker container?

[Mats Bengtsson]

On 2020-01-22 11:36, David Kastrup wrote:

Han-Wen Nienhuys  writes:
A posiitive: all major platforms run on x86 64-bit CPUs, and there are
Open Source VM solutions for OSX and Windows. So, a VM running Linux
on OSX or Windows can be almost as fast as running Linux
directly. This makes GUB's cross compilation to other architectures
superfluous.
In ancient times, I've had to be the guinea pig for VMware in a large
company and the experience was not thrilling: two of the typical
problems were that not the full power of a 2-CPU computer was made
available to Linux, only part of the RAM (and the basic RAM was getting
managed by Windows) and mapping the file system through a Windows file
system was really bad since Linux is so much faster in that regard.
That particularly concerned Git use which really depends on fast file
system operations.  Creating a virtual disk helped somewhat, creating an
actual partition for Linux helped quite more.  At some point of time I
started booting that partition, and, well, the experiment did not
deliver a lot more relevant data afterwards...

I have no experience with Docker and containers.  That was a full VM at
the time.


I recently installed Ubuntu within the Windows Subsystem for Linux in 
Windows 10 and installed the standard Linux package of Lilypond therein, 
which runs very smoothly. Our sys. admins don't give us admin rights for 
our Win 10 installation, so this is the first time for many years that 
I've been able to run Lilypond on that computer. For the same reason, I 
haven't been able to benchmark the processing speed against the Win 
version of Lilypond, but at least it runs significantly faster than on 
my old Linux machine, so it cannot be too bad.


However, just as any other solution based on Docker or other VMs, it's 
not aimed at the average Windows user, especially if you want to run 
Frescobaldi or any other GUI based editor, since Windows WSL doesn't 
support X applications natively.


   /Mats

Re: Context paths (and the Edition Engraver)

[Jan-Peter Voigt]

Am 21.01.20 um 18:50 schrieb Dan Eble:
> On Jan 21, 2020, at 11:31, Jan-Peter Voigt  wrote:
>> I'd like that, though it would be a quite invasive change.
>> And if we stay with the string for the context id and then use
>> lists/paths in the \context statement like
>> \new Staff = "choir" << \new Voice = "soprano" …
>>
>> and then use
>> \context Voice = choir.soprano
>>
>> it would be inconsistent with \new  = "…"
>>
>> I will write down some more text about this topic later.
>
> I see similarities with languages like CSS and XPATH select nodes in a DOM.  
> Notation borrowed directly from them will not integrate well into LilyPond, 
> but it might be fruitful to ask how we could modify expressions like these to 
> fit in.
>
> %% find the voice in the example quoted above, very specifically
> \context Staff#choir > Voice#soprano { … } % CSS
> \context Staff[@id=choir]/Voice[@id=soprano] { … } % XPATH
>
> %% ditto, but using context type only
> \context Staff > Voice { … }   % CSS
> \context Staff/Voice { … } % XPATH
>
> %% ditto, but using ID only
> \context #choir > #soprano { … }   % CSS
> \context [@id=choir]/[@id=soprano] { … }   % XPATH
>
> %% find the context where the rehearsalMark property is defined
> \context [rehearsalMark] { … } % CSS
> \context [@rehearsalMark] { … }% XPATH
> —
> Dan
>

I am amazed what kind of discussion is raised on this topic :)

I'd suggest alternative commands to create something like an
XQuery/CSS/whatever functionality. Elsewhere in this thread David (K)
answered to syntax ideas that would break the current model.

To have the possibility to address contexts *like* in CSS has some
appeal. But IMHO it shouldn't disturb the current input scheme. So
alternate commands might help here.
Perhaps *like*:

\getContext the.path.to.the.context

... or ...

\getContext \thisContext."..".otherContextsName

*don't know if implementing \thisContext is trivial*

The idea that an ID of a context is a list and not a string does attract
me, but I see a major change that has to be done very carefully.

If the ID is a string (or symbol) the path can be easily constructed
with ly:context-parent.
Something like:
%%
\version "2.19.83"

logContextPath = \applyContext
#(lambda (context)
   (define (context-path context)
 (let ((parent (ly:context-parent context)))
   (if (ly:context? parent)
   `(,@(context-path parent) ,(ly:context-id context))
   (list (ly:context-id context)))
   ))
   (ly:message "context path: ~A" (context-path context)))

\new Score = "mymusic" {
  \new StaffGroup = "choir" <<
\new Staff = "soprano" <<
  { c''4 \logContextPath } \\
  { g'4 \logContextPath }
>>
  >>
}
%%


Just some thoughts.

I hope to be able to write some more text about it soon.
But I have another task that will take up a lot of my time in the coming
weeks. You are welcome to ask me questions which I will try to answer.
But I will not be able to be active for the next weeks.

Jan-Peter

Re: patches without tracker issues

[James]

On 22.01.2020 11:06, Han-Wen Nienhuys wrote:
these were sent before I had access, I think. Can you folksk have  a 
look?


https://codereview.appspot.com/559370043/
https://codereview.appspot.com/581470047/


I'll follow up



---
Regards

James

Re: lily: fix some type conversion warnings (issue 557190043 by hanw...@gmail.com)

[benko . pal]

On 2020/01/22 10:15:03, hanwenn wrote:
> Why do we need to have the distinction between size_t and int? I know
the
> standard library returns size_t in some places, but is there any
reason for
> LilyPond to used unsigned integers anywhere? 

It's not just the sign (btw I like unsigned integers):
they also differ in size, at least on 64 bit architectures
int is int32_t, size_t is uint64_t.

p

https://codereview.appspot.com/557190043/

Re: GUILE 2/3 and string encoding cost

[David Kastrup]

Han-Wen Nienhuys  writes:

> I looked a bit through the GUILE source code to see what is going on.
>
> I believe our current hypothesis (LilyPond's slowdown is caused by
> expensive unicode transcoding into 32-bit strings) is incorrect.
>
> If you look into the source code, you can see that the UTF-8 -> SCM
> conversion checks if there are any code points over 255
>
>
> https://git.savannah.nongnu.org/cgit/guile.git//tree/libguile/strings.c/?id=1b8e9ca0e37fab366435436995248abdfc780a10#n1620
>
> if there aren't, it uses Latin1 encoding ("narrow == 1") to encode the
> string as a normal byte array. This code walks the string twice, but that
> is very cheap due to CPU cache locality, so it should be
> essentially equivalent to whatever GUILE 1.8 was doing.

GUILE 1.8 did not walk the string even once.

> LilyPond internally doesn't use any Unicode strings, as all our
> identifiers are pure ascii, as well as internal strings (eg. font
> glyph names). This means that files that do not use Unicode characters
> at all should have the same overhead for strings as GUILE 1.8.

We already use the latin1 calls for LilyPond internals.

> Even so, if the input flie does use UTF-8, there should be little
> overhead, because the number of texts that we process is always
> small. LilyPond is not a text processor.
>
> So, what hard data do we have on GUILE 2/3 slowness, and what does
> that data say?

That data says "humongous slowdown".  There is not much more than
speculation what this is caused by as far as I know.

-- 
David Kastrup

Re: Context paths (and the Edition Engraver)

[David Kastrup]

Flaming Hakama by Elaine  writes:

>> -- Forwarded message --
>> From: David Kastrup 
>> To: Dan Eble 
>> Cc: lilypond-devel@gnu.org
>> Bcc:
>> Date: Tue, 21 Jan 2020 22:51:29 +0100
>> Subject: Re: Context paths (and the Edition Engraver)
>> Dan Eble  writes:
>>
>> > On Jan 21, 2020, at 14:37, David Kastrup  wrote:
>> >>
>> >> StaffGroup = "organ" . Staff = "upper" . Voice . SubVoice = 2
>> >
>> > OK.  It would be an understandable growth on the current face of
>> LilyPond. :)
>> >
>> > Questions follow, but I'm not asking you to spend time investigating.
>> >
>> > Do you think we could achieve making the quotes optional for some
>> > simple IDs, and making the whitespace optional?
>> >
>> > StaffGroup=organ.Staff=upper.Voice.SubVoice=2
>>
>> It would all be optional (except in \lyricsmode or \markup) but I am
>> skeptical that it would make for a great convention.
>>
>> > In a situation where the user didn't care to constrain the context
>> > types, do you think could they be omitted, or would we have to invent
>> > a placeholder?
>> >
>> > =organ.=upper..=2
>> > X=organ.X=upper.X.X=2
>>
>> I don't think that this would be a reasonable syntax.
>
> Just chiming in to agree that this syntax is not very clear.
> At least, I find it confusing to see what looks like multiple assignments
> (use of =) on the same line.
> The = in this case is not being used in this case to define something, but
> to identify something that presumably already exists.
>
>
> Is there any hope of having syntax like one of the following?
>
> StaffGroup("organ").Staff("upper").Voice.SubVoice("2")
> StaffGroup["organ"].Staff["upper"].Voice.SubVoice["2"]

Not as much "hope" as "horror".  This does not fit at all into current
LilyPond syntax.

I mean, something like

"[" = <>[
"(" = <>(

{ \new Voice[c'8 d' e'] (g' c''2) }

is valid code.  Not a good idea at all, but stopping the parser from
working with it in order to give it a different meaning would cause a
whole lot of wreckage.

And what LilyPond can reasonably parse in the context of music functions
falls into comparatively narrow classes and requires a lot of planning
and magic behind the scenes.

-- 
David Kastrup

Re: Packages/modules

[Urs Liska]

Am Mittwoch, den 22.01.2020, 11:06 +0100 schrieb David Kastrup:
> Urs Liska  writes:
> 
> > Am Dienstag, den 21.01.2020, 11:19 +0100 schrieb Urs Liska:
> > > > Ok.  One thing to think about is that we want package files to
> > > > be
> > > > contributed by "ordinary" users.  But something like
> > > > 
> > > > \exportSymbols transposeSequence,instrumentGroup,scratchMyBack
> > > > 
> > > > would be perfectly feasible syntactical sugar.
> > > > 
> > > 
> > > I'll be more verbose than probably necessary, just to make sure
> > > we're
> > > talking about the same thing.
> > > 
> > > ...
> > > 
> > > If I got you right then from my experience with openLilyLib and
> > > creating project environments (which basically are the same as
> > > packages), I would say:
> > > 
> > >  * I'm all for hiding names in packages by default and having to
> > >explicitly expose/export the package's interface
> > >  
> > 
> > One more implication: If variables and functions have to be
> > explicitly
> > exported it will be easier for external tools (like Frescobaldi) to
> > add
> > proper support for extensions.
> > 
> > I assume that at one point Frescobaldi will
> > 
> >  * know about available (core and external) extensions
> >  * provide ways to "use" an extension (as part of the Score wizard
> > and
> >locally)
> >  * at that point know about the options that can be passed to that
> >package
> >  * provide autocompletion and highlighting for available symbols
> >exported from extensions
> >  * provide actions to generate the code for getting and setting
> > package
> >options
> > 
> > So when planning the syntax of that export it would be good to take
> > the
> > needs/interest of IDEs into account that will not work with the
> > result
> > as LilyPond does but that parse the package files themselves.
> 
> Maybe we should have single-command exports after all

You mean that a package has to export every function or variable
separately? I think that would be good wrt self-documentation.

Would it be possible to export some meta information too? e.g. the type
of a variable, the signature (including if it is a music-function etc.)
of a function? That would be good.
The options do something like that already in their current
implementation. The \registerOption is used as a guard against trying
\setOption with an unregistered option, but the concept can directly be
used for package documentation.

>  and have a
> (non-optional ?) documentation string to be used as mouse-over?  I
> think
> a unified approach to documentation might go a long way towards basic
> usability.

A non-optional docstring sounds nice. This itself doesn't guarantee
that the docstrings are actually *helpful*, but if e.g. Frescobaldi
provides direct access to them pressure would certainly mount to
improve them one by one.

Wilbert plans (I don't recall if he said so in a presentation or just
while chatting) to use his current rewriting of Frescobaldi's LilyPond
parser to provide such "introspection", through a mouse-over effect or
by extending the interface of the autocompletion dropdown. This will
*certainly* make it more straightforward to get used to LilyPond's
internals.

Urs

> 

Re: packaging lilypond as a docker container?

[David Kastrup]

Han-Wen Nienhuys  writes:

> Let me explain my thinking more in detail. (I've CC'd Jan in case he wants
> to give further historical insights)
>
> The objective of GUB was to provide installers for LilyPond with the
> following requirements:
>
> 1) build for all platforms (OSX, Windows, Linux) with minimal effort,
> without having to manage windows and OSX machines.
> 2) all platforms use exactly the same versions of all software, so everyone
> has the same experience, and problems always reproduce across platforms.
>
> 1) implied setting up a cross compiling environment. This is a pain in the
> ass, because GCC must be compiled for each (host, target) pair separately.
> This is very slow and takes up a lot of space.

[Skipping over a number of details]

> In the intervening 15 years, many things have changed.
>
> A negative: apple licensing restrictions say that we can't cross-compile to
> OSX (although to be honest, I never read the terms & conditions for the
> XCode stuff I downloaded from apple back in the day. Maybe we've always
> been in violation of some T)

I think the "no crosscompilation" thing was some Jobs vendetta against
something specific he didn't like, and was introduced to thwart one
thing in particular.  I don't remember the details, but it's possible
that the ancient XCode is from before that event.  We have tried a bit
of finding out but unsuccessfully so.

A possibility would be to go to whatever became of OpenDarwin and use
that, but as far as I remember some of the font handling libraries in
use were actually tied to MacOSX in particular (Cocoa by now?).

> A posiitive: all major platforms run on x86 64-bit CPUs, and there are
> Open Source VM solutions for OSX and Windows. So, a VM running Linux
> on OSX or Windows can be almost as fast as running Linux
> directly. This makes GUB's cross compilation to other architectures
> superfluous.

In ancient times, I've had to be the guinea pig for VMware in a large
company and the experience was not thrilling: two of the typical
problems were that not the full power of a 2-CPU computer was made
available to Linux, only part of the RAM (and the basic RAM was getting
managed by Windows) and mapping the file system through a Windows file
system was really bad since Linux is so much faster in that regard.
That particularly concerned Git use which really depends on fast file
system operations.  Creating a virtual disk helped somewhat, creating an
actual partition for Linux helped quite more.  At some point of time I
started booting that partition, and, well, the experiment did not
deliver a lot more relevant data afterwards...

I have no experience with Docker and containers.  That was a full VM at
the time.

> A positive: Linux now supports namespaces, and in particular: file
> system namespaces. This makes it possible to build a native Linux
> package fully isolated from its host system, thus skipping GUB's
> laborious GCC setup.  This also lets you run a regular package manager
> on the namespaced file system, and this is essentially what Docker
> gives you: a way to install arbitrary set of dependencies and use
> them. This makes GUB's package manager superfluous.
>
> A positive: Docker is open source, and extremely popular
> technology. Docker does the hard work of packaging up VM + Docker for
> windows and OSX as installer, and providing a management/distribution
> mechanism for images.
>
> All in all, I think we could use Docker for distributing
> LilyPond. Then we would have a distribution mechanism that works for
> OSX and Windows, and it has no overhead to compiling LilyPond as
> normal Linux package. Because we can control the base image, we can
> precisely control the dependencies, and be sure that the binary
> behaves the same across all OSes. The best is that building releases
> won't require any arcane knowledge of cross-compilation quirks.

It suffers from the "someone needs to do it" syndrome and I have
absolutely no idea how smooth such executables will install and behave
compared to native ones on MacOSX and Windows.  I'd have to rely on
evaluation by the respective OS users to have an idea about that.

> GUIX is Jan's current project. I think it has some similarities to
> GUB, but it is focused on providing an environment where all binaries
> are reproducibly built from source. This is much more ambitious than
> GUB, and seems overkill compared to what we need for LilyPond. I think
> using it also entails many more compilation steps, which would makes
> the release process slow again.

I don't think it has an answer for the elephant in the room: Windows.

-- 
David Kastrup

Re: packaging lilypond as a docker container?

[Jonas Hahnfeld via Discussions on LilyPond development]

Let me respond to some of your points:

Am Mittwoch, den 22.01.2020, 11:00 +0100 schrieb Han-Wen Nienhuys:
> Let me explain my thinking more in detail. (I've CC'd Jan in case he wants to 
> give further historical insights) 
> 
> 
> The objective of GUB was to provide installers for LilyPond with the 
> following requirements:
> 
> 1) build for all platforms (OSX, Windows, Linux) with minimal effort, without 
> having to manage windows and OSX machines.
> 
> 2) all platforms use exactly the same versions of all software, so everyone 
> has the same experience, and problems always reproduce across platforms.
> 
> 1) implied setting up a cross compiling environment. This is a pain in the 
> ass, because GCC must be compiled for each (host, target) pair separately. 
> This is very slow and takes up a lot of space.

And doesn't even work. Did you recently try to run the binaries
produced for FreeBSD? They're dynamically linked to GUB's glibc which
isn't packaged up. Not surprisingly, FreeBSD has its own libc with
different so-versions and so on.

> However, the major reason why this is a PITA is that many pieces of software 
> don't support cross-compiling well. In particular, some software runs itself 
> as part of the build (lilypond is an example, but so do Python and Fontconfig 
> IIRC). Making them cross-compile is something that Jan did in his 
> professional career (embedded software!), so he knew how to do it for 
> LilyPond. It is however very intricate, and time consuming to test. One has 
> to pore over compilation logs, understand what all these packages are trying 
> to do, and then come up with ways to do the same in a cross-compiling manner. 
> This is why the GUB specifications are patching compilation steps in creative 
> ways.

The patches are horrible to maintain once you want to bump more than
just to a new patch version.

> I found it demoralizing to work on, because I want to work on music 
> typesetting, and not on patching up someone else's wonky compilation script.
> 
> Because we now have a very expensive and complicated setup (GCC !), it makes 
> sense to utilize this setup across builds. For example, if we upgrade the 
> lilypond version, we'd rather not have to rebuild GCC from scratch. So, GUB 
> has a packaging system, where you can upgrade LilyPond (or any intermediate 
> dependency) without starting from scratch.

I do the same for the dependencies of LilyPond, except for all that
complicated packaging stuff. If you screw something up, just rebuild
it. This takes around ~20 minutes on my Laptop from 2016.

> We started out with a set of shell scripts, but they were hard to maintain, 
> so we built GUB. So, to me, Jonas' proposal to have a set of simple 
> compilation scripts is a little funny.

As a predecessor of GUB, this probably involved cross-compilation? I'm
not doing this, it's just native compilation all way long.

> Cross compiling has these problems above, but the "native" compile also has 
> problems. One has ensure that the native compile doesn't pick up dependencies 
> from the system it is compiled from, as that could cause unpredictable 
> behavior. You'd run GUB on a machine that is used for LilyPond, so all of the 
> dependencies can be consumed from both GUB and the host machine, and it is 
> easy for something to go wrong here.

I think this got much better, and I looks like I finally convinced pkg-
config not to use anything from the system. This only leaves the
compiler headers and libc. All of the rest is built as static libraries
which end up in the LilyPond executable.

> In the intervening 15 years, many things have changed. 
> 
> A negative: apple licensing restrictions say that we can't cross-compile to 
> OSX (although to be honest, I never read the terms & conditions for the XCode 
> stuff I downloaded from apple back in the day. Maybe we've always been in 
> violation of some T)
> 
> A posiitive: all major platforms run on x86 64-bit CPUs, and there are Open 
> Source VM solutions for OSX and Windows. So, a VM running Linux on OSX or 
> Windows can be almost as fast as running Linux directly. This makes GUB's 
> cross compilation to other architectures superfluous.
> 
> A positive: Linux now supports namespaces, and in particular: file system 
> namespaces. This makes it possible to build a native Linux package fully 
> isolated from its host system, thus skipping GUB's laborious GCC setup. This 
> also lets you run a regular package manager on the namespaced file system, 
> and this is essentially what Docker gives you: a way to install arbitrary set 
> of dependencies and use them. This makes GUB's package manager superfluous.
> 
> A positive: Docker is open source, and extremely popular technology. Docker 
> does the hard work of packaging up VM + Docker for windows and OSX as 
> installer, and providing a management/distribution mechanism for images.
> 
> All in all, I think we could use Docker for distributing LilyPond. Then we 
> would have 

Re: lily: fix some type conversion warnings (issue 557190043 by hanw...@gmail.com)

[hanwenn]

On 2020/01/22 10:15:03, hanwenn wrote:
> On 2020/01/21 15:21:56, Dan Eble wrote:
> >
>
https://codereview.appspot.com/557190043/diff/581490044/lily/pointer-group-interface.cc
> > File lily/pointer-group-interface.cc (right):
> > 
> >
>
https://codereview.appspot.com/557190043/diff/581490044/lily/pointer-group-interface.cc#newcode30
> > lily/pointer-group-interface.cc:30: return arr ? int (arr->size ())
: 0;
> > MHO: Return a vsize.  Somewhere, sometime, someone will probably
say,
> > 
> > my_things_.size () < pgi.count ()
> > 
> > and the compiler will warn about comparing int to size_t.
> > 
> > I don't suggest that you switch to vsize and ferret out all the
consequences
> at
> > this time if you don't want to invest your time in that, but that
it's better
> to
> > leave this line alone and live with the warning until there is time
to take a
> > proper look.
> > 
> >
https://codereview.appspot.com/557190043/diff/581490044/lily/quote-iterator.cc
> > File lily/quote-iterator.cc (right):
> > 
> >
>
https://codereview.appspot.com/557190043/diff/581490044/lily/quote-iterator.cc#newcode104
> > lily/quote-iterator.cc:104: int hi = int (scm_c_vector_length
(vec));
> > If scm_c_vector_length () is returning size_t, then why don't we
work in terms
> > of size_t?  Any place I see a cast, especially a C-style cast, I
feel the urge
> > to stop and ask, "Why is this OK?"  So I'd rather not see them where
they can
> be
> > avoided.
> > 
> > https://codereview.appspot.com/557190043/diff/581490044/lily/stem.cc
> > File lily/stem.cc (right):
> > 
> >
https://codereview.appspot.com/557190043/diff/581490044/lily/stem.cc#newcode94
> > lily/stem.cc:94: int len = int (scm_ilength (lst)); // -1 for dotted
lists!
> > ditto
> 
> Can we go back to basics? 
> 
> Why do we need to have the distinction between size_t and int? I know
the
> standard library returns size_t in some places, but is there any
reason for
> LilyPond to used unsigned integers anywhere? 
> 
> I think the most practical solution is to cast any unsigned to int
directly
> where we get it out of the external library.

alternately, we could just do -Wno-conversion in the makefile, and we
wouldn't have to do any work here at all.

https://codereview.appspot.com/557190043/

Re: lily: fix some type conversion warnings (issue 557190043 by hanw...@gmail.com)

[hanwenn]

Reviewers: dan_faithful.be, lemzwerg, Dan Eble,

Message:
On 2020/01/21 15:21:56, Dan Eble wrote:
>
https://codereview.appspot.com/557190043/diff/581490044/lily/pointer-group-interface.cc
> File lily/pointer-group-interface.cc (right):
> 
>
https://codereview.appspot.com/557190043/diff/581490044/lily/pointer-group-interface.cc#newcode30
> lily/pointer-group-interface.cc:30: return arr ? int (arr->size ()) :
0;
> MHO: Return a vsize.  Somewhere, sometime, someone will probably say,
> 
> my_things_.size () < pgi.count ()
> 
> and the compiler will warn about comparing int to size_t.
> 
> I don't suggest that you switch to vsize and ferret out all the
consequences at
> this time if you don't want to invest your time in that, but that it's
better to
> leave this line alone and live with the warning until there is time to
take a
> proper look.
> 
>
https://codereview.appspot.com/557190043/diff/581490044/lily/quote-iterator.cc
> File lily/quote-iterator.cc (right):
> 
>
https://codereview.appspot.com/557190043/diff/581490044/lily/quote-iterator.cc#newcode104
> lily/quote-iterator.cc:104: int hi = int (scm_c_vector_length (vec));
> If scm_c_vector_length () is returning size_t, then why don't we work
in terms
> of size_t?  Any place I see a cast, especially a C-style cast, I feel
the urge
> to stop and ask, "Why is this OK?"  So I'd rather not see them where
they can be
> avoided.
> 
> https://codereview.appspot.com/557190043/diff/581490044/lily/stem.cc
> File lily/stem.cc (right):
> 
>
https://codereview.appspot.com/557190043/diff/581490044/lily/stem.cc#newcode94
> lily/stem.cc:94: int len = int (scm_ilength (lst)); // -1 for dotted
lists!
> ditto

Can we go back to basics? 

Why do we need to have the distinction between size_t and int? I know
the standard library returns size_t in some places, but is there any
reason for LilyPond to used unsigned integers anywhere? 

I think the most practical solution is to cast any unsigned to int
directly where we get it out of the external library.


 


Description:
lily: fix some type conversion warnings

Please review this at https://codereview.appspot.com/557190043/

Affected files (+10, -11 lines):
  M lily/break-substitution.cc
  M lily/lookup.cc
  M lily/multi-measure-rest-engraver.cc
  M lily/page-spacing-result.cc
  M lily/pointer-group-interface.cc
  M lily/quote-iterator.cc
  M lily/stem.cc


Index: lily/break-substitution.cc
diff --git a/lily/break-substitution.cc b/lily/break-substitution.cc
index 
901f6b0ceb188bcf470b8333ec825d0761aaf3a6..73379ba395af1e99c02b06858d08a3077255f7e7
 100644
--- a/lily/break-substitution.cc
+++ b/lily/break-substitution.cc
@@ -113,13 +113,12 @@ again:
 }
   else if (scm_is_vector (src))
 {
-  int len = scm_c_vector_length (src);
+  size_t len = scm_c_vector_length (src);
   SCM nv = scm_c_make_vector (len, SCM_UNDEFINED);
-  for (int i = 0; i < len; i++)
+  for (size_t i = 0; i < len; i++)
 {
-  SCM si = scm_from_int (i);
-  scm_vector_set_x (nv, si,
-do_break_substitution (scm_vector_ref (src, si)));
+  scm_c_vector_set_x (nv, i,
+  do_break_substitution (scm_c_vector_ref (src, 
i)));
 }
 }
   else if (scm_is_pair (src))
Index: lily/lookup.cc
diff --git a/lily/lookup.cc b/lily/lookup.cc
index 
c7b8cafcc39dcda4204804e03201755dcce20768..117fe96d609b1f283dc11cbb8b4605aba8035d0e
 100644
--- a/lily/lookup.cc
+++ b/lily/lookup.cc
@@ -311,7 +311,7 @@ Lookup::round_filled_polygon (vector const ,
 
   for (vsize i = 0; i < points.size (); i++)
 {
-  int i0 = i;
+  int i0 = (int) i;
   int i1 = (i + 1) % points.size ();
   int i2 = (i + 2) % points.size ();
   Offset p0 = points[i0];
Index: lily/multi-measure-rest-engraver.cc
diff --git a/lily/multi-measure-rest-engraver.cc 
b/lily/multi-measure-rest-engraver.cc
index 
0e8d694cd5a342a5156b350f2cd8a4fc4ea987df..8901cd4d171c6d1da5a98e1c98cc9f9a5988f5cd
 100644
--- a/lily/multi-measure-rest-engraver.cc
+++ b/lily/multi-measure-rest-engraver.cc
@@ -142,7 +142,7 @@ Multi_measure_rest_engraver::initialize_grobs ()
   Spanner *sp = make_spanner ("MultiMeasureRestScript", e->self_scm 
());
   make_script_from_event (sp, context (),
   e->get_property ("articulation-type"),
-  i);
+  int(i));
   SCM dir = e->get_property ("direction");
   if (is_direction (dir))
 sp->set_property ("direction", dir);
Index: lily/page-spacing-result.cc
diff --git a/lily/page-spacing-result.cc b/lily/page-spacing-result.cc
index 
106c7e882295df3f1e99f0fc7971f888acf65d15..bfb25286912ac8229cba54977d89904d53063dcd
 100644
--- a/lily/page-spacing-result.cc
+++ b/lily/page-spacing-result.cc
@@ -41,7 +41,7 @@ Page_spacing_result::average_force () const
   for (vsize i = 0; i < page_count (); i++)
 

Re: packaging lilypond as a docker container?

[David Kastrup]

Han-Wen Nienhuys  writes:

> On Tue, Jan 21, 2020 at 11:28 AM David Kastrup  wrote:
>
>> But the elephant in the room is Windows.  Han-Wen says he never wants to
>> touch GUB again (and he actually didn't as far as I remember) but I

Something happened between brain and keyboard.  What I meant to write
was "and he actually didn't as far as I remember during the time of my
active tenure on LilyPond".  I agree that this does come off quite wrong
and will be misleading at the very least to everyone who hasn't stuck
around long enough to be versed in LilyPond's long-term history.

Sorry for that.

>> 
>>
>
> excuse me?
>
> $ git log  |grep ^Author | sort | uniq -c|sort -n |tail
>  41 Author: PhilHolmes 
> 108 Author: Masamichi Hosoda 
> 109 Author: lilytest 
> 160 Author: Masamichi Hosoda 
> 169 Author: Han-Wen Nienhuys 
> 482 Author: janneke 
> 569 Author: Han-Wen Nienhuys 
> 899 Author: hanwen 
> 912 Author: hanwen 
>2801 Author: Jan Nieuwenhuizen 

-- 
David Kastrup

patches without tracker issues

[Han-Wen Nienhuys]

these were sent before I had access, I think. Can you folksk have  a look?

https://codereview.appspot.com/559370043/
https://codereview.appspot.com/581470047/

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: Packages/modules

[David Kastrup]

Urs Liska  writes:

> Am Dienstag, den 21.01.2020, 11:19 +0100 schrieb Urs Liska:
>> > Ok.  One thing to think about is that we want package files to be
>> > contributed by "ordinary" users.  But something like
>> > 
>> > \exportSymbols transposeSequence,instrumentGroup,scratchMyBack
>> > 
>> > would be perfectly feasible syntactical sugar.
>> > 
>> 
>> I'll be more verbose than probably necessary, just to make sure we're
>> talking about the same thing.
>> 
>> ...
>>
>> If I got you right then from my experience with openLilyLib and
>> creating project environments (which basically are the same as
>> packages), I would say:
>> 
>>  * I'm all for hiding names in packages by default and having to
>>explicitly expose/export the package's interface
>>  
>
> One more implication: If variables and functions have to be explicitly
> exported it will be easier for external tools (like Frescobaldi) to add
> proper support for extensions.
>
> I assume that at one point Frescobaldi will
>
>  * know about available (core and external) extensions
>  * provide ways to "use" an extension (as part of the Score wizard and
>locally)
>  * at that point know about the options that can be passed to that
>package
>  * provide autocompletion and highlighting for available symbols
>exported from extensions
>  * provide actions to generate the code for getting and setting package
>options
>
> So when planning the syntax of that export it would be good to take the
> needs/interest of IDEs into account that will not work with the result
> as LilyPond does but that parse the package files themselves.

Maybe we should have single-command exports after all and have a
(non-optional ?) documentation string to be used as mouse-over?  I think
a unified approach to documentation might go a long way towards basic
usability.

-- 
David Kastrup

Re: packaging lilypond as a docker container?

[Han-Wen Nienhuys]

Let me explain my thinking more in detail. (I've CC'd Jan in case he wants
to give further historical insights)

The objective of GUB was to provide installers for LilyPond with the
following requirements:

1) build for all platforms (OSX, Windows, Linux) with minimal effort,
without having to manage windows and OSX machines.
2) all platforms use exactly the same versions of all software, so everyone
has the same experience, and problems always reproduce across platforms.

1) implied setting up a cross compiling environment. This is a pain in the
ass, because GCC must be compiled for each (host, target) pair separately.
This is very slow and takes up a lot of space.

However, the major reason why this is a PITA is that many pieces of
software don't support cross-compiling well. In particular, some software
runs itself as part of the build (lilypond is an example, but so do Python
and Fontconfig IIRC). Making them cross-compile is something that Jan did
in his professional career (embedded software!), so he knew how to do it
for LilyPond. It is however very intricate, and time consuming to test. One
has to pore over compilation logs, understand what all these packages are
trying to do, and then come up with ways to do the same in a
cross-compiling manner. This is why the GUB specifications are patching
compilation steps in creative ways.

I found it demoralizing to work on, because I want to work on music
typesetting, and not on patching up someone else's wonky compilation script.

Because we now have a very expensive and complicated setup (GCC !), it
makes sense to utilize this setup across builds. For example, if we upgrade
the lilypond version, we'd rather not have to rebuild GCC from scratch. So,
GUB has a packaging system, where you can upgrade LilyPond (or any
intermediate dependency) without starting from scratch.

We started out with a set of shell scripts, but they were hard to maintain,
so we built GUB. So, to me, Jonas' proposal to have a set of simple
compilation scripts is a little funny.

Cross compiling has these problems above, but the "native" compile also has
problems. One has ensure that the native compile doesn't pick up
dependencies from the system it is compiled from, as that could cause
unpredictable behavior. You'd run GUB on a machine that is used for
LilyPond, so all of the dependencies can be consumed from both GUB and the
host machine, and it is easy for something to go wrong here.

In the intervening 15 years, many things have changed.

A negative: apple licensing restrictions say that we can't cross-compile to
OSX (although to be honest, I never read the terms & conditions for the
XCode stuff I downloaded from apple back in the day. Maybe we've always
been in violation of some T)

A posiitive: all major platforms run on x86 64-bit CPUs, and there are Open
Source VM solutions for OSX and Windows. So, a VM running Linux on OSX or
Windows can be almost as fast as running Linux directly. This makes GUB's
cross compilation to other architectures superfluous.

A positive: Linux now supports namespaces, and in particular: file system
namespaces. This makes it possible to build a native Linux package fully
isolated from its host system, thus skipping GUB's laborious GCC setup.
This also lets you run a regular package manager on the namespaced file
system, and this is essentially what Docker gives you: a way to install
arbitrary set of dependencies and use them. This makes GUB's package
manager superfluous.

A positive: Docker is open source, and extremely popular technology. Docker
does the hard work of packaging up VM + Docker for windows and OSX as
installer, and providing a management/distribution mechanism for images.

All in all, I think we could use Docker for distributing LilyPond. Then we
would have a distribution mechanism that works for OSX  and Windows, and it
has no overhead to compiling LilyPond as normal Linux package. Because we
can control the base image, we can precisely control the dependencies, and
be sure that the binary behaves the same across all OSes. The best is that
building releases won't require any arcane knowledge of cross-compilation
quirks.

GUIX is Jan's current project. I think it has some similarities to GUB, but
it is focused on providing an environment where all binaries are
reproducibly built from source. This is much more ambitious than GUB, and
seems overkill compared to what we need for LilyPond. I think using it also
entails many more compilation steps, which would makes the release process
slow again.


On Tue, Jan 21, 2020 at 11:28 AM David Kastrup  wrote:

> Jonas Hahnfeld  writes:
>
> > Am Dienstag, den 21.01.2020, 02:38 -0600 schrieb Karlin High:
> >> On 1/21/2020 1:49 AM, Han-Wen Nienhuys wrote:
> >> > if GUB is used, who is maintaining and/or working on it?
> >>
> >> Almost exactly a year ago, there was a sizable "Please test gub" effort
> >> initiated by Knut Petersen.
> >>
> >> <
> >> 

Re: switching to Python 3.x

[Jonas Hahnfeld]

Am Montag, den 06.01.2020, 19:12 +0100 schrieb Jonas Hahnfeld:
> Am Donnerstag, den 19.12.2019, 20:13 +0100 schrieb Jonas Hahnfeld:
> > Hello friends of Python 3!
> > 
> > to make the initial proposal short: With today's patches, I think
> > 'master' would be ready to switch over to Python 3.x.
> > As a reminder, Python 2 will go EOL on 1st of January 2020 - in 13 days
> > if I counted correctly. That probably doesn't mean that Python 2.x will
> > go away immediately, but I strongly believe that active projects should
> > switch. Eventually Linux distributions will start dropping packages
> > that depend on Python 2.x, and I personally don't want LilyPond to be
> > one of them.
> > 
> > There's still one preparatory patch left, but it will break
> > compatibility with Python 2.4 which is still supported right now:
> > https://sourceforge.net/p/testlilyissues/issues/5645/
> > 
> > 
> > As far as I can tell, the change works fine with Python 2.7. We could
> > land this immediately before switching to Python 3 if building 'master'
> > with Python 2.4 is critical. If needed, we would still retain the
> > ability to bisect via Python 2.7.
> > 
> > The switch itself requires 3 changes which I would propose to squash
> > into one commit: (I'll put all of this into 
> > https://sourceforge.net/p/testlilyissues/issues/5646/
> > 
> > )
> > 1) Adapt the build system to find and require Python 3.
> > patch: 
> > https://codereview.appspot.com/545370043
> > 
> > 
> > 2) The largest part of the switch is running 2to3 which is now able to
> > handle the rest of the conversion automatically. For reference, changes
> > for current 'master' are here: 
> > https://codereview.appspot.com/573340043
> > 
> > 
> > 3) Fix-up two places in the scripts afterwards:
> > 3a) Remove the call to sys.setdefaultencoding which doesn't exist in
> > Python 3.x (and is only available right now because of a dirty hack).
> > 3b) Replace cgi.escape by html.escape. While not strictly needed, I
> > think we should include this change because they removed cgi.escape in
> > Python 3.8 after only deprecating it in Python 3.2. The replacement
> > html.escape is only available since that Python 3.2, so we can't do
> > this before switching.
> > patch: 
> > https://codereview.appspot.com/561270043
> > 
> > 
> > 
> > One point worth discussing is the future minimum version of Python 3.x.
> > In the patch, I'm proposing Python 3.5 because it will allow us to
> > address a few deprecation warnings, especially about module 'imp'.
> > It should be available in most Linux distributions:
> >  * CentOS 7 provides Python 3.6 since some minor releases
> >  * Debian 9 (Stretch) has packages for Python 3.5, Debian 10 (Buster)
> > even for Python 3.6
> >  * Ubuntu 16.04 LTS has Python 3.5, version 18.04 LTS has Python 3.6
> > 
> > Debian 8 (Jessie) only has Python 3.4, but even its LTS support will
> > end next June (2020-06). Would it be acceptable to drop support for
> > this distribution building 'master'?
> > On the GUB side, I already added a spec for Python 3.7.4 (also for
> > Windows via binary packages) and this worked successfully in September.
> > Is there a major distribution missing that doesn't provide at least
> > Python 3.5?
> > 
> > Let me know what you think!
> 
> So far, I've only received a single (positive) response off-list and a
> bit of feedback on the posted patches. What do others think?
> To make this explicit: The proposal is to drop support for Python 2
> (now EOL), requiring everyone wishing to build LilyPond 'master' to
> have an appropriate version of Python 3 available. This should be
> sufficiently easy (see above), but I'd like to have consensus on this.

Another two weeks have passed without much participation in this thread
(a single response by David on GUB). Not sure if this topic was
discussed in Salzburg?
To make progress I went ahead and removed the needs_wait on 
https://sourceforge.net/p/testlilyissues/issues/5646/. Please note that
Issue #5645 is gone after I discovered some edge cases where the code
is still needed for Python 2.7.

If you want to test, please either apply the three patches on Rietveld
or alternatively checkout dev/hahnjo/python3.

Hoping for some comments this time,
Jonas


signature.asc
Description: This is a digitally signed message part

Re: document and test slur score debugging (issue 555160043 by hanw...@gmail.com)

[lemzwerg--- via Discussions on LilyPond development]

Some minor nits...


https://codereview.appspot.com/555160043/diff/549440043/lily/slur.cc
File lily/slur.cc (right):

https://codereview.appspot.com/555160043/diff/549440043/lily/slur.cc#newcode495
lily/slur.cc:495: "Slurs are formatted by a number of combination of
left/right "
s/combination/combinations/

And please replace all trailing blanks in the documentation string with
leading ones.

https://codereview.appspot.com/555160043/diff/549440043/lily/slur.cc#newcode496
lily/slur.cc:496: "end-point, and then picking the slur with the lowest
demerit score. "
s/end-point/end points/

https://codereview.appspot.com/555160043/diff/549440043/lily/slur.cc#newcode498
lily/slur.cc:498: "(ie. note heads) in the direction in half space
increments until we "
s/ie./i.e.,/ (the final comma is for the sake of PDF output)

https://codereview.appspot.com/555160043/

Re: Encoding changes for Python 3 (issue 571280043 by jonas.hahnf...@gmail.com)

[jonas . hahnfeld]

Reviewers: lemzwerg,

Message:
Closing as the change got smaller and smaller as I discovered more edge
cases where the current code matters with Python 2.x

Description:
Encoding changes for Python 3

This is required to eventually make the scripts work with Python 3.
The change happens to also work with Python 2.7 (and maybe also 2.6),
but most certainly not with Python 2.5 or older supported up to now.

Please review this at https://codereview.appspot.com/571280043/

Affected files (+16, -27 lines):
  M python/lilylib.py
  M python/midi.py
  M python/musicexp.py
  M python/musicxml.py
  M python/utilities.py
  M scripts/midi2ly.py
  M scripts/musicxml2ly.py


Index: python/lilylib.py
diff --git a/python/lilylib.py b/python/lilylib.py
index 
821d88ce4e68c40f00d2f597bb3637f41b16c52a..114e48877c28e988b5b100767feb1103505f0270
 100644
--- a/python/lilylib.py
+++ b/python/lilylib.py
@@ -17,6 +17,7 @@
 # along with LilyPond.  If not, see .
 
 import __main__
+import codecs
 import glob
 import os
 import re
@@ -25,6 +26,10 @@ import sys
 import optparse
 import time
 
+# Reload sys to get sys.setdefaultencoding back.
+reload(sys)
+sys.setdefaultencoding ('utf-8')
+
 
 # Users of python modules should include this snippet
 # and customize variables below.
@@ -50,23 +55,9 @@ except:
 return p
 underscore = _
 
-# Urg, Python 2.4 does not define stderr/stdout encoding
-# Maybe guess encoding from LANG/LC_ALL/LC_CTYPE?
-
-reload (sys)
-sys.setdefaultencoding ('utf-8')
-import codecs
-sys.stdout = codecs.getwriter ('utf8') (sys.stdout)
-sys.stderr = codecs.getwriter ('utf8') (sys.stderr)
-
 def encoded_write(f, s):
 f.write (s.encode (f.encoding or 'utf-8', 'replace'))
 
-# ugh, Python 2.5 optparse requires Unicode strings in some argument
-# functions, and refuse them in some other places
-def display_encode (s):
-return s.encode (sys.stderr.encoding or 'utf-8', 'replace')
-
 # Lilylib globals.
 program_version = '@TOPLEVEL_VERSION@'
 program_name = os.path.basename (sys.argv[0])
@@ -118,7 +109,7 @@ def is_verbose ():
 return is_loglevel ("DEBUG")
 
 def stderr_write (s):
-encoded_write (sys.stderr, s)
+sys.stderr.write (s)
 
 def print_logmessage (level, s, fullmessage = True, newline = True):
 if (is_loglevel (level)):
Index: python/midi.py
diff --git a/python/midi.py b/python/midi.py
index 
f4d17eae71ce918420658da50fa358bdad6d072e..1eca6c11a1921c2e37aac85be55e93e591bf0182
 100644
--- a/python/midi.py
+++ b/python/midi.py
@@ -176,19 +176,19 @@ def _parse_tracks (midi, pos, num_tracks, clocks_max):
 if num_tracks > 256:
 raise error('too many tracks: %d' % num_tracks)
 for i in xrange(num_tracks):
-trackdata, pos = _parse_hunk (midi, pos, 'track', 'MTrk')
+trackdata, pos = _parse_hunk (midi, pos, 'track', b'MTrk')
 yield list (_parse_track_body (trackdata, clocks_max))
 # if pos < len(midi):
 # warn
 
 def parse_track (track, clocks_max=None):
-track_body, end = _parse_hunk (track, 0, 'track', 'MTrk')
+track_body, end = _parse_hunk (track, 0, 'track', b'MTrk')
 # if end < len(track):
 # warn
 return list (_parse_track_body (track_body, clocks_max))
 
 def parse (midi, clocks_max=None):
-header, first_track_pos = _parse_hunk(midi, 0, 'file', 'MThd')
+header, first_track_pos = _parse_hunk(midi, 0, 'file', b'MThd')
 try:
 format, num_tracks, division = struct.unpack ('>3H', header[:6])
 except struct.error:
Index: python/musicexp.py
diff --git a/python/musicexp.py b/python/musicexp.py
index 
b3905af634d6ebef4fa9bc6fd52fad6132f9b401..7fc35eea3029d40c145a44fde1bac28cd88b16b4
 100644
--- a/python/musicexp.py
+++ b/python/musicexp.py
@@ -818,7 +818,7 @@ class Lyrics:
 for l in self.lyrics_syllables:
 lstr += l
 #lstr += "\n}"
-return lstr.encode('utf-8')
+return lstr
 
 class Header:
 
Index: python/musicxml.py
diff --git a/python/musicxml.py b/python/musicxml.py
index 
2a456f4c9e6172176f58c5de820207281091e365..4841358f492bcc93ea204e5b2393225ff7628d1d
 100644
--- a/python/musicxml.py
+++ b/python/musicxml.py
@@ -41,7 +41,7 @@ class Xml_node(object):
 if not self._children:
 return ''
 
-return ''.join([c.get_text() for c in self._children]).encode('utf-8')
+return ''.join([c.get_text() for c in self._children])
 
 def message(self, msg):
 ly.warning(msg)
Index: python/utilities.py
diff --git a/python/utilities.py b/python/utilities.py
index 
e124539d3bf2b6ecfe5e0c136d9a73ce2da72758..4513142fbb9983554d8585db85ea48e23538410e
 100644
--- a/python/utilities.py
+++ b/python/utilities.py
@@ -63,8 +63,6 @@ def hex_to_color(hex_val):
 return None
 
 def split_string_and_preserve_doublequoted_substrings(value):
-if isinstance(value, unicode):
-value = value.encode('utf-8')
 import shlex
 

Re: packaging lilypond as a docker container?

[Han-Wen Nienhuys]

On Tue, Jan 21, 2020 at 11:28 AM David Kastrup  wrote:

> But the elephant in the room is Windows.  Han-Wen says he never wants to
> touch GUB again (and he actually didn't as far as I remember) but I
>

excuse me?

$ git log  |grep ^Author | sort | uniq -c|sort -n |tail
 41 Author: PhilHolmes 
108 Author: Masamichi Hosoda 
109 Author: lilytest 
160 Author: Masamichi Hosoda 
169 Author: Han-Wen Nienhuys 
482 Author: janneke 
569 Author: Han-Wen Nienhuys 
899 Author: hanwen 
912 Author: hanwen 
   2801 Author: Jan Nieuwenhuizen 

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: document and test slur score debugging (issue 555160043 by hanw...@gmail.com)

[hanwenn]

On 2020/01/21 14:44:04, Dan Eble wrote:
>
https://codereview.appspot.com/555160043/diff/577340043/input/regression/slur-inspect-quants.ly
> File input/regression/slur-inspect-quants.ly (right):
> 
>
https://codereview.appspot.com/555160043/diff/577340043/input/regression/slur-inspect-quants.ly#newcode3
> input/regression/slur-inspect-quants.ly:3: texidoc = "This file
demonstrates the
> slur scoring debug functionality"
> What is the expected result?  Is the visual output important?  Is it
enough that
> this doesn't crash?  These are questions I would ask if this case
appeared in my
> regtest diffs.

added some more description.

https://codereview.appspot.com/555160043/

GUILE 2/3 and string encoding cost

[Han-Wen Nienhuys]

I looked a bit through the GUILE source code to see what is going on.

I believe our current hypothesis (LilyPond's slowdown is caused by
expensive unicode transcoding into 32-bit strings) is incorrect.

If you look into the source code, you can see that the UTF-8 -> SCM
conversion checks if there are any code points over 255


https://git.savannah.nongnu.org/cgit/guile.git//tree/libguile/strings.c/?id=1b8e9ca0e37fab366435436995248abdfc780a10#n1620

if there aren't, it uses Latin1 encoding ("narrow == 1") to encode the
string as a normal byte array. This code walks the string twice, but that
is very cheap due to CPU cache locality, so it should be
essentially equivalent to whatever GUILE 1.8 was doing.

The conversion in the other direction is here:
https://git.savannah.nongnu.org/cgit/guile.git//tree/libguile/strings.c/?id=1b8e9ca0e37fab366435436995248abdfc780a10#n2065

as you can see, if the string is narrow (Latin1/ASCII), it uses the cheap
path as well.

LilyPond internally doesn't use any Unicode strings, as all our identifiers
are pure ascii, as well as internal strings (eg. font glyph names). This
means that files that do not use Unicode characters at all should have the
same overhead for strings as GUILE 1.8.

Even so, if the input flie does use UTF-8, there should be little overhead,
because the number of texts that we process is always small. LilyPond is
not a text processor.

So, what hard data do we have on GUILE 2/3 slowness, and what does that
data say?

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: Packages/modules

[Urs Liska]

Am Dienstag, den 21.01.2020, 11:19 +0100 schrieb Urs Liska:
> > Ok.  One thing to think about is that we want package files to be
> > contributed by "ordinary" users.  But something like
> > 
> > \exportSymbols transposeSequence,instrumentGroup,scratchMyBack
> > 
> > would be perfectly feasible syntactical sugar.
> > 
> 
> I'll be more verbose than probably necessary, just to make sure we're
> talking about the same thing.
> 
> ...
>
> If I got you right then from my experience with openLilyLib and
> creating project environments (which basically are the same as
> packages), I would say:
> 
>  * I'm all for hiding names in packages by default and having to
>explicitly expose/export the package's interface
>  

One more implication: If variables and functions have to be explicitly
exported it will be easier for external tools (like Frescobaldi) to add
proper support for extensions.

I assume that at one point Frescobaldi will

 * know about available (core and external) extensions
 * provide ways to "use" an extension (as part of the Score wizard and
   locally)
 * at that point know about the options that can be passed to that
   package
 * provide autocompletion and highlighting for available symbols
   exported from extensions
 * provide actions to generate the code for getting and setting package
   options

So when planning the syntax of that export it would be good to take the
needs/interest of IDEs into account that will not work with the result
as LilyPond does but that parse the package files themselves.

Urs