Re: Python 3 (was: 2.20 where are we?)
On Mon, Sep 30, 2019 at 5:46 PM Joram wrote: > Am 27.09.19 um 22:34 schrieb Matthew Peveler: > > long vs int, unicode vs str, StringIO vs io, iter.next vs > > iter.__next__, reload, xrange vs range. > > It is very well feasible to support both version. I hope by shims you > mean something like¹ > > from __future__ import division, print_function > from builtins import range > > because there is no need to customly write those. I consider the link > quite helpful. > > Cheers > > > ¹ https://python-future.org/compatible_idioms.html Yes for using __future__, but no for builtins while it would make things easier, it is not something that is actually built into python and would be installable for PyPI via the future package ( https://pypi.org/project/future/). So I had to hand define the shims, though I mostly went for python2 -> python3 shim such: if sys.version_info > (3,): xrange = range raw_input = input as I dislike overwriting builtin functions/variables if I can avoid it as it leads to potentially surprising behavior, though reversing that would not be very difficult if so desired. Matt ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Python 3 (was: 2.20 where are we?)
Am 27.09.19 um 22:34 schrieb Matthew Peveler: > long vs int, unicode vs str, StringIO vs io, iter.next vs > iter.__next__, reload, xrange vs range. It is very well feasible to support both version. I hope by shims you mean something like¹ from __future__ import division, print_function from builtins import range because there is no need to customly write those. I consider the link quite helpful. Cheers ¹ https://python-future.org/compatible_idioms.html ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Python 3 (was: 2.20 where are we?)
On Sep 27, 2019, at 16:34, Matthew Peveler wrote: > > I'll have another email later on with patches for having this branch run > under both python2.7 & 3 as the necessary backport efforts were not really > all that extravagant with just a handful of shims around the changes you > noted in long vs int, unicode vs str, StringIO vs io, iter.next vs > iter.__next__, reload, xrange vs range. Are these complications desirable? A clean and obvious implementation requiring Python 3 will be easier to maintain. — Dan ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Python 3 (was: 2.20 where are we?)
You are right, that was added in my original debug effort. Attached is the second patch without the added print. It looks like the files are getting stored as `.bin` files, though are just plain text when you open them. If someone could explain to me the proper way to attach patches here (or maybe it's just gmail being funny?), I would appreciate it. I'll have another email later on with patches for having this branch run under both python2.7 & 3 as the necessary backport efforts were not really all that extravagant with just a handful of shims around the changes you noted in long vs int, unicode vs str, StringIO vs io, iter.next vs iter.__next__, reload, xrange vs range. The primary thing missing is making the stepmake/stepmake/python-module-*.make files work for the range of versions, but I'm afraid that is out of my wheel house of ability. Matt On Thu, Sep 26, 2019 at 6:03 AM Jonas Hahnfeld wrote: > Am Montag, den 23.09.2019, 12:26 -0300 schrieb Matthew Peveler: > > On Sat, Sep 21, 2019 at 5:52 AM Jonas Hahnfeld via lilypond-devel < > lilypond-devel@gnu.org> wrote: > > > On top of that I've worked on the attached patches which brings the > make targets check / test and doc back to life with Python 3.7.4. > > > > In applying your patches and running "make check", I encountered a > couple of errors in scripts/build/output-distance.py, which would be summed > up as: > > > > 1. Python 2 old style classes had a default __cmp__ which would compare > the id() of the classes, and used implicitly for sort. Python 3 removed > this, but need to define a __lt__ method for sort compat. > > 2. attempting to open midi files as str files, though they are filled > with binary data (and python/midi.py seems to expect binary in interacting > with the contents). > > > > Please see the attached for small patches for these two things, and got > "make check" to run for me using Python 3.7.4 as well. > > > > Regards, > > Matt > > Ha, I already wondered why there are two targets check and test that > both ran into the same issues (when I started). Turns out, the target > "check" runs "test" but not the other way around, and I only verified > that "test" works in the end... > > The patches make sense to me and also work on my system. You should > probably remove the print() statement in the second patch, though. > > Thanks, > Jonas > From 298f17d82948335b98ffcf2feb9c14c80ddad390 Mon Sep 17 00:00:00 2001 From: Matthew Peveler Date: Mon, 23 Sep 2019 06:15:27 -0300 Subject: [PATCH 2/2] Fix py3 compat issues in output_distance.py --- scripts/build/output-distance.py | 9 +++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/scripts/build/output-distance.py b/scripts/build/output-distance.py index 83999e4559..7e31959ee4 100755 --- a/scripts/build/output-distance.py +++ b/scripts/build/output-distance.py @@ -150,6 +150,9 @@ class GrobSignature(object): self.bbox = (bbox_x, bbox_y) self.centroid = (bbox_x[0] + bbox_x[1], bbox_y[0] + bbox_y[1]) +def __lt__ (self, other): +return id(self) < id(other) + def __repr__ (self): return '%s: (%.2f,%.2f), (%.2f,%.2f)\n' % (self.name, self.bbox[0][0], @@ -355,6 +358,9 @@ class FileLink(object): self._distance = None self.file_names = (f1, f2) +def __lt__ (self, other): +return id(self) < id(other) + def text_record_string (self): return '%-30f %-20s\n' % (self.distance (), self.name () @@ -568,7 +574,7 @@ class MidiFileLink (TextFileCompareLink): def get_content (self, oldnew): import midi -data = FileCompareLink.get_content (self, oldnew) +data = open (oldnew, 'rb').read () midi = midi.parse (data) tracks = midi[1] @@ -1381,4 +1387,3 @@ def main (): if __name__ == '__main__': main () - -- 2.23.0 ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Python 3 (was: 2.20 where are we?)
Am Montag, den 23.09.2019, 12:26 -0300 schrieb Matthew Peveler: > On Sat, Sep 21, 2019 at 5:52 AM Jonas Hahnfeld via lilypond-devel > wrote: > > On top of that I've worked on the attached patches which brings the make > > targets check / test and doc back to life with Python 3.7.4. > > In applying your patches and running "make check", I encountered a couple of > errors in scripts/build/output-distance.py, which would be summed up as: > > 1. Python 2 old style classes had a default __cmp__ which would compare the > id() of the classes, and used implicitly for sort. Python 3 removed this, but > need to define a __lt__ method for sort compat. > 2. attempting to open midi files as str files, though they are filled with > binary data (and python/midi.py seems to expect binary in interacting with > the contents). > > Please see the attached for small patches for these two things, and got "make > check" to run for me using Python 3.7.4 as well. > > Regards, > Matt Ha, I already wondered why there are two targets check and test that both ran into the same issues (when I started). Turns out, the target "check" runs "test" but not the other way around, and I only verified that "test" works in the end... The patches make sense to me and also work on my system. You should probably remove the print() statement in the second patch, though. Thanks, Jonas signature.asc Description: This is a digitally signed message part ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Python 3 (was: 2.20 where are we?)
On Mon, Sep 23, 2019 at 12:26 PM Matthew Peveler wrote: > Please see the attached for small patches for these two things, and got "make check" to run for me using Python 3.7.4 as well. Hm, not sure why the patch files got converted to binary when sending, though I've also made them available at https://gist.github.com/MasterOdin/8141656b872682a3a540f7af4a1c0d9f for good measure. Matt > 0001-add-__lt__-method-for-py2-old-style-class-sort-compa.patch Description: Binary data 0002-read-midi-files-as-binary.patch Description: Binary data ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Python 3 (was: 2.20 where are we?)
On Sat, Sep 21, 2019 at 5:52 AM Jonas Hahnfeld via lilypond-devel < lilypond-devel@gnu.org> wrote: > On top of that I've worked on the attached patches which brings the make targets check / test and doc back to life with Python 3.7.4. In applying your patches and running "make check", I encountered a couple of errors in scripts/build/output-distance.py, which would be summed up as: 1. Python 2 old style classes had a default __cmp__ which would compare the id() of the classes, and used implicitly for sort. Python 3 removed this, but need to define a __lt__ method for sort compat. 2. attempting to open midi files as str files, though they are filled with binary data (and python/midi.py seems to expect binary in interacting with the contents). Please see the attached for small patches for these two things, and got "make check" to run for me using Python 3.7.4 as well. Regards, Matt 0001-add-__lt__-method-for-py2-old-style-class-sort-compa.patch Description: Binary data 0002-read-midi-files-as-binary.patch Description: Binary data ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: 2.20 where are we?
Andrew Bernard writes: > So let me get this straight. > > It's not the case that GUB is completely broken. We can still build > releases. It's not broken _yet_ because Python 2 is still more or less available and of course we can keep using it (ignoring security issues) as long as we want. This will hit the native builds first. There have been some hits concerning Ghostscript that are currently in check but may come back. Also some other issues. > DK is working steadily to cherry pick items for 2.20. I'd not use the label "steadily" here. > Python 2 to Python 3 is a major issue. > > So, I offered to do the 2->3 port a long time ago but circumstances > prevented me from doing so. Would it be constructive if I launched > into that aspect? I cant understand the lilypond internals code but I > have extensive Python experience. > > Would this be helpful to moving forward? > > But have people already started on this, I thought? Kurt had started on it but asked for help getting more done. I don't know to what degrees others have actually stepped in also, but it's not like one could not share work by arranging who ports what. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Python 3 (was: 2.20 where are we?)
Am Samstag, den 21.09.2019, 12:09 +1000 schrieb Andrew Bernard:
> So let me get this straight.
>
> It's not the case that GUB is completely broken. We can still build
> releases.
>
> DK is working steadily to cherry pick items for 2.20.
>
> Python 2 to Python 3 is a major issue.
>
> So, I offered to do the 2->3 port a long time ago but circumstances
> prevented me from doing so. Would it be constructive if I launched into
> that aspect? I cant understand the lilypond internals code but I have
> extensive Python experience.
>
> Would this be helpful to moving forward?
>
> But have people already started on this, I thought?
I've also started looking into this and used the branch
dev/knupero/lilypy3devel as a starting point (see also
https://lists.gnu.org/archive/html/bug-lilypond/2019-08/msg00014.html).
On top of that I've worked on the attached patches which brings the
make targets check / test and doc back to life with Python 3.7.4. Maybe
they can be added to the branch mentioned above to serve as a single
source of truth? I know the third patch is pretty large, let me know if
I should try to split it up...
As I'm pretty new to the development of Lilypond, I'm not really sure
if there's something else to do in the source code repository itself?
I'm pretty sure I didn't get to run through all error branches in all
scripts, but the targets mentioned in the documentation work for me
right now. If not, sounds like working on GUB would be next?
Regards,
Jonas
From f6595ce68dc47e4e60f94dd8b80dbd5573e4c360 Mon Sep 17 00:00:00 2001
From: Jonas Hahnfeld
Date: Thu, 19 Sep 2019 20:26:25 +0200
Subject: [PATCH 1/5] lilylib: sys.stderr.write expects strings
... not encoded bytes. This fixes the first error when running the
check / test target.
---
python/lilylib.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python/lilylib.py b/python/lilylib.py
index 3850f4e2a3..ef156115a5 100644
--- a/python/lilylib.py
+++ b/python/lilylib.py
@@ -127,7 +127,7 @@ def is_verbose ():
return is_loglevel ("DEBUG")
def stderr_write (s):
-encoded_write (sys.stderr, s)
+sys.stderr.write (s)
def print_logmessage (level, s, fullmessage = True, newline = True):
if (is_loglevel (level)):
--
2.23.0
From 5630b85ec8d9a7344009a221916ba87904287848 Mon Sep 17 00:00:00 2001
From: Jonas Hahnfeld
Date: Thu, 19 Sep 2019 19:56:24 +0200
Subject: [PATCH 2/5] Read linking files in binary mode
This fixes the check / test target up to musicxml.
---
python/book_snippets.py | 11 +--
1 file changed, 5 insertions(+), 6 deletions(-)
diff --git a/python/book_snippets.py b/python/book_snippets.py
index b84c269e6a..0a2f3fbb31 100644
--- a/python/book_snippets.py
+++ b/python/book_snippets.py
@@ -645,12 +645,11 @@ printing diff against existing file." % filename)
try:
if (self.global_options.use_source_file_names
and isinstance (self, LilypondFileSnippet)):
-fout = open (dst, 'w')
-fin = open (src, 'r')
-for line in fin.readlines ():
-fout.write (line.replace (self.basename (), self.final_basename ()))
-fout.close ()
-fin.close ()
+content = open (src, 'rb').read ()
+basename = self.basename ().encode ('utf-8')
+final_basename = self.final_basename ().encode ('utf-8')
+content = content.replace (basename, final_basename)
+open (dst, 'wb').write (content)
else:
try:
os.link (src, dst)
--
2.23.0
From 99a7c48d8c331f5a0a61884905c140ec654b3b8f Mon Sep 17 00:00:00 2001
From: Jonas Hahnfeld
Date: Wed, 18 Sep 2019 21:56:02 +0200
Subject: [PATCH 3/5] Fix musicxml2ly with Python 3
This passes the targets doc and check / test.
---
python/book_snippets.py | 24 +
python/musicexp.py | 29 ++--
python/musicxml.py | 20 +++---
python/rational.py | 13 ++---
python/utilities.py | 2 +-
scripts/musicxml2ly.py | 60 -
6 files changed, 76 insertions(+), 72 deletions(-)
diff --git a/python/book_snippets.py b/python/book_snippets.py
index 0a2f3fbb31..0f7888bcb9 100644
--- a/python/book_snippets.py
+++ b/python/book_snippets.py
@@ -605,7 +605,8 @@ class LilypondSnippet (Snippet):
if self.relevant_contents (existing) != self.relevant_contents (self.full_ly ()):
warning ("%s: duplicate filename but different contents of original file,\n\
printing diff against existing file." % filename)
-ly.stderr_write (self.filter_pipe (self.full_ly (), 'diff -u %s -' % filename))
+diff_against_existing = self.filter_pipe (self.full_ly ().encode ('utf-8'), 'diff -u %s -' % filename)
+ly.stderr_write
Re: 2.20 where are we?
Am 21. September 2019 04:09:36 MESZ schrieb Andrew Bernard : >So let me get this straight. > >It's not the case that GUB is completely broken. We can still build >releases. > >DK is working steadily to cherry pick items for 2.20. > >Python 2 to Python 3 is a major issue. > >So, I offered to do the 2->3 port a long time ago but circumstances >prevented me from doing so. Would it be constructive if I launched into > >that aspect? I cant understand the lilypond internals code but I have >extensive Python experience. > >Would this be helpful to moving forward? > This may be an even more important way to help than with the teo Frescobaldi topics I suggested to you. >But have people already started on this, I thought? > IIRC there were some things done or st least planned but surely nothing that should prevent you from joining ... Best Urs > >Andrew > > > >___ >lilypond-devel mailing list >lilypond-devel@gnu.org >https://lists.gnu.org/mailman/listinfo/lilypond-devel -- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet. ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
2.20 where are we?
So let me get this straight. It's not the case that GUB is completely broken. We can still build releases. DK is working steadily to cherry pick items for 2.20. Python 2 to Python 3 is a major issue. So, I offered to do the 2->3 port a long time ago but circumstances prevented me from doing so. Would it be constructive if I launched into that aspect? I cant understand the lilypond internals code but I have extensive Python experience. Would this be helpful to moving forward? But have people already started on this, I thought? Andrew ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
