Re: python: fix installation of .py files when building in separate dir (issue 549810043 by hanw...@gmail.com)

[jonas . hahnfeld]

I could probably be happy with the following:
diff --git a/make/lilypond-vars.make b/make/lilypond-vars.make
index 3f1e992626..3ac59f19a4 100644
--- a/make/lilypond-vars.make
+++ b/make/lilypond-vars.make
@@ -18,6 +18,8 @@ endif
 LANGS=$(shell $(PYTHON) $(top-src-dir)/python/langdefs.py)
 
 export PYTHONPATH:=$(top-src-dir)/python:$(PYTHONPATH)
+# Don't create __pycache__ in the source directory.
+export PYTHONDONTWRITEBYTECODE=1
 
 the-script-dir=$(wildcard $(script-dir))
 
diff --git a/python/GNUmakefile b/python/GNUmakefile
index b541206df3..efe9329038 100644
--- a/python/GNUmakefile
+++ b/python/GNUmakefile
@@ -2,7 +2,7 @@ depth = ..
 
 SUBDIRS=auxiliar
 
-STEPMAKE_TEMPLATES=install-out po
+STEPMAKE_TEMPLATES=install-out install po
 
 PY_MODULES_IN = $(call src-wildcard,*.py)
 OUT_PY_MODULES = $(PY_MODULES_IN:%=$(outdir)/%)
@@ -17,13 +17,11 @@ $(outdir)/%.py: %.py
 
 default: $(OUT_PY_MODULES)
 
-INSTALLATION_OUT_SUFFIXES = 1 2
+INSTALLATION_DIR = $(local_lilypond_datadir)/python
+INSTALLATION_FILES = $(filter-out $(call src-wildcard, *_test.py),
$(PY_MODULES_IN))
 
-INSTALLATION_OUT_DIR1=$(local_lilypond_datadir)/python
-INSTALLATION_OUT_FILES1=$(PY_MODULES_IN)
-
-INSTALLATION_OUT_DIR2 = $(local_lilypond_datadir)/python/__pycache__
-INSTALLATION_OUT_FILES2 = $(wildcard $(outdir)/__pycache__/*.pyc)
+INSTALLATION_OUT_DIR = $(local_lilypond_datadir)/python/__pycache__
+INSTALLATION_OUT_FILES = $(wildcard $(outdir)/__pycache__/*.pyc)
 
 local-test: book_base_test.py
$(PYTHON) $<

Afterwards I can submit a change to avoid copying the .py files over and
have them only in the source directory:
diff --git a/python/GNUmakefile b/python/GNUmakefile
index efe9329038..731fadb8e5 100644
--- a/python/GNUmakefile
+++ b/python/GNUmakefile
@@ -5,17 +5,16 @@ SUBDIRS=auxiliar
 STEPMAKE_TEMPLATES=install-out install po
 
 PY_MODULES_IN = $(call src-wildcard,*.py)
-OUT_PY_MODULES = $(PY_MODULES_IN:%=$(outdir)/%)
 
 include $(depth)/make/stepmake.make
 
-$(outdir)/%.py: %.py
+$(outdir)/%.pyc.dummy: %.py
$(call ly_progress,Making,$@,(py compile))
-   cp $< $@
-   PYTHONOPTIMIZE= $(PYTHON) -c 'import py_compile;
py_compile.compile ("$@", doraise=True)'
-   chmod 755 $@
+   # Do not use buildscript-dir, this has not been traversed yet.
+   $(PYTHON) $(top-src-dir)/scripts/build/compile.py $<
+   touch $@
 
-default: $(OUT_PY_MODULES)
+default: $(PY_MODULES_IN:%.py=$(outdir)/%.pyc.dummy)
 
 INSTALLATION_DIR = $(local_lilypond_datadir)/python
 INSTALLATION_FILES = $(filter-out $(call src-wildcard, *_test.py),
$(PY_MODULES_IN))
diff --git a/scripts/build/compile.py b/scripts/build/compile.py
new file mode 100644
index 00..294abbaf4a
--- /dev/null
+++ b/scripts/build/compile.py
@@ -0,0 +1,14 @@
+#!@PYTHON@
+
+from importlib.util import cache_from_source
+from os.path import basename
+import py_compile
+import sys
+
+if len (sys.argv) != 2:
+print ('Usage: compile.py ')
+sys.exit (1)
+
+src = sys.argv[1]
+cfile = cache_from_source (basename(src))
+py_compile.compile (src, cfile=cfile, doraise=True)

https://codereview.appspot.com/549810043/

Re: python: fix installation of .py files when building in separate dir (issue 549810043 by hanw...@gmail.com)

[jonas . hahnfeld]

On 2020/04/06 19:51:11, hanwenn wrote:
> suppress __pycache__

The suppression only works for the test. Running a full build still
gives me python/__pycache__/ in the source directory because PYTHONPATH
points there.

https://codereview.appspot.com/549810043/

Re: Install python modules from build directory (issue 575960043 by jonas.hahnf...@gmail.com)

[jonas . hahnfeld]

https://codereview.appspot.com/575960043/diff/557650043/make/lilypond-vars.make
File make/lilypond-vars.make (right):

https://codereview.appspot.com/575960043/diff/557650043/make/lilypond-vars.make#newcode20
make/lilypond-vars.make:20: export
PYTHONPATH:=$(top-build-dir)/python/$(outconfbase):$(PYTHONPATH)
On 2020/04/06 19:50:35, hanwenn wrote:
> On 2020/04/05 21:54:40, hahnjo wrote:
> > On 2020/04/05 21:48:46, hanwenn wrote:
> > > This will potentially make the out-www build fail. It's much
better to load
> > the
> > > source files directly, because they're always there and they're
always up to
> > > date. 
> > 
> > I don't see why this should break: It's now back to the state it was
before
> you
> > changed it. But I can change it to out/ if you prefer.
> 
> Loading .py files from out/ directories is an antipattern in the
lilypond build
> that makes working on the scripts a PITA, because the dependency
tracking
> doesn't work, and you have to remember to rebuild them by hand. Please
don't
> reintroduce this.

I don't really get that argument. As far as I understand, your recent
change ensures that python/ is built first to get the modules before
they are needed. AFAICT this dependency works correctly, and if you
touch a .py file in the source directory it is correctly copied over and
byte-compiled. Maybe you can describe what's an "antipattern" about
this?

https://codereview.appspot.com/575960043/diff/557650043/python/GNUmakefile
File python/GNUmakefile (right):

https://codereview.appspot.com/575960043/diff/557650043/python/GNUmakefile#newcode27
python/GNUmakefile:27: local-test: default
On 2020/04/06 19:50:35, hanwenn wrote:
> On 2020/04/05 21:54:40, hahnjo wrote:
> > On 2020/04/05 21:48:46, hanwenn wrote:
> > > if you do it like this, this should depend on
$(outdir)/book_base_test.py
> and
> > > the rule should use $< . Otherwise, editing the test file doesn't
redo the
> > copy.
> > 
> > No, default depends on $(OUT_PY_MODULES). And we really need all
modules
> because
> > book_base_test.py includes other files.
> 
> you're right.  I do think that this is not the way to go to suppress
the
> __pycache__ directory. If that is a concern, we should just pass -B to
the
> commandline.

It doesn't surpress __pycache__, it just uses the already compiled
modules in the build directory. This way there is exactly one location
where the cache is.

https://codereview.appspot.com/575960043/

Re: Install python modules from build directory (issue 575960043 by jonas.hahnf...@gmail.com)

[hanwenn]

https://codereview.appspot.com/575960043/diff/557650043/make/lilypond-vars.make
File make/lilypond-vars.make (right):

https://codereview.appspot.com/575960043/diff/557650043/make/lilypond-vars.make#newcode20
make/lilypond-vars.make:20: export
PYTHONPATH:=$(top-build-dir)/python/$(outconfbase):$(PYTHONPATH)
On 2020/04/05 21:54:40, hahnjo wrote:
> On 2020/04/05 21:48:46, hanwenn wrote:
> > This will potentially make the out-www build fail. It's much better
to load
> the
> > source files directly, because they're always there and they're
always up to
> > date. 
> 
> I don't see why this should break: It's now back to the state it was
before you
> changed it. But I can change it to out/ if you prefer.

Loading .py files from out/ directories is an antipattern in the
lilypond build that makes working on the scripts a PITA, because the
dependency tracking doesn't work, and you have to remember to rebuild
them by hand. Please don't reintroduce this.

https://codereview.appspot.com/575960043/diff/557650043/python/GNUmakefile
File python/GNUmakefile (right):

https://codereview.appspot.com/575960043/diff/557650043/python/GNUmakefile#newcode27
python/GNUmakefile:27: local-test: default
On 2020/04/05 21:54:40, hahnjo wrote:
> On 2020/04/05 21:48:46, hanwenn wrote:
> > if you do it like this, this should depend on
$(outdir)/book_base_test.py and
> > the rule should use $< . Otherwise, editing the test file doesn't
redo the
> copy.
> 
> No, default depends on $(OUT_PY_MODULES). And we really need all
modules because
> book_base_test.py includes other files.

you're right.  I do think that this is not the way to go to suppress the
__pycache__ directory. If that is a concern, we should just pass -B to
the commandline.

https://codereview.appspot.com/575960043/

Re: GUB failure

[Phil Holmes]

I bit the bullet and started the upgrade to 16.04.  I check that's OK then 
consider 18.04 later.  Upgrading via the GUI software updater.


--
Phil Holmes


- Original Message - 
From: "Thomas Morley" 

To: "Phil Holmes" 
Cc: "David Kastrup" ; "Devel" 
Sent: Monday, April 06, 2020 4:18 PM
Subject: Re: GUB failure


Am Mo., 6. Apr. 2020 um 16:52 Uhr schrieb Phil Holmes 
:


As predicted, GUB has failed to build on my system.  The error message 
shows

that it's because I have Python 3.4.3 and I need at least 3.5.  If anyone
knows how to upgrade my Ubuntu 14 machine to the later Python version,
please say.


Probably
http://ubuntuhandbook.org/index.php/2017/07/install-python-3-6-1-in-ubuntu-16-04-lts/
adapted to your needs may help.


Otherwise I'll have to upgrade to Ubuntu 16, which I was
avoiding while I still had a working installation.


_If_ you need to upgrade, why not newest Ubuntu?

I'll test GUB with current master on my Ubuntu 18.04, though results
likely tomorrow...

Cheers,
 Harm

Re: GUB failure

[Thomas Morley]

Am Mo., 6. Apr. 2020 um 16:52 Uhr schrieb Phil Holmes :
>
> As predicted, GUB has failed to build on my system.  The error message shows
> that it's because I have Python 3.4.3 and I need at least 3.5.  If anyone
> knows how to upgrade my Ubuntu 14 machine to the later Python version,
> please say.

Probably
http://ubuntuhandbook.org/index.php/2017/07/install-python-3-6-1-in-ubuntu-16-04-lts/
adapted to your needs may help.

> Otherwise I'll have to upgrade to Ubuntu 16, which I was
> avoiding while I still had a working installation.

_If_ you need to upgrade, why not newest Ubuntu?

I'll test GUB with current master on my Ubuntu 18.04, though results
likely tomorrow...

Cheers,
  Harm

Re: GUB failure

[Matthew Peveler]

On Mon, Apr 6, 2020 at 11:52 PM Phil Holmes  wrote:

> If anyone knows how to upgrade my Ubuntu 14 machine to the later Python
> version,
> please say.


You can use the deadsnakes ppa (fkrull/deadsnakes
 for pre
Ubuntu-16.04, deadsnakes/ppa
 otherwise) to
install non-included versions of python on Ubuntu distros:sudo

add-apt-repository --yes ppa:fkrull/deadsnakes
sudo apt-get update
sudo apt-get install --yes python3.5 python3.5-dev

You will have to manually install pip I believe doing:
wget https://bootstrap.pypa.io/get-pip.py
sudo python3.5 get-pip.py

Re: GUB failure

[David Kastrup]

"Phil Holmes"  writes:

> As predicted, GUB has failed to build on my system.  The error message
> shows that it's because I have Python 3.4.3 and I need at least 3.5.
> If anyone knows how to upgrade my Ubuntu 14 machine to the later
> Python version, please say.  Otherwise I'll have to upgrade to Ubuntu
> 16, which I was avoiding while I still had a working installation.

Well, I think the LTS installations at that time were supported for
5 years.  I am not sure that upgrading to 16 would still work.  The
usual is to do

sudo update-manager

and follow suggestions for how to upgrade distributions.  There is also
the more explicit

sudo update-manager -d

but it tends to offer updates to the newest release (possibly
prerelease) rather than the latest stable.

-- 
David Kastrup

GUB failure

[Phil Holmes]

As predicted, GUB has failed to build on my system.  The error message shows 
that it's because I have Python 3.4.3 and I need at least 3.5.  If anyone 
knows how to upgrade my Ubuntu 14 machine to the later Python version, 
please say.  Otherwise I'll have to upgrade to Ubuntu 16, which I was 
avoiding while I still had a working installation.


--
Phil Holmes

Re: Problem with LSR import

[Werner LEMBERG]

> (btw, the html->texinfo conversion doesn’t appear to turn  into
> @itemize bullet as it ideally ought to.)

If time permits I'll continue work on my LSR import script that I've
already posted to the list.  It uses `pandoc` for the HTML to texinfo
conversion, which seems to work quite well.


Werner

Re: Fix line comments for new snippets (issue 549820043 by d...@gnu.org)

[lemzwerg--- via Discussions on LilyPond development]

> I'd rather not have 2.21.0 in a state inconsistent
> with its makelsr program.  So if I were to push,
> I'd need to push the results of a new makelsr run
> as well.

OK, I misunderstood.



https://codereview.appspot.com/549820043/

Re: Fix line comments for new snippets (issue 549820043 by d...@gnu.org)

[dak]

Reviewers: lemzwerg,

Message:
On 2020/04/06 12:40:50, lemzwerg wrote:
> LGTM, and please push directly.

I'd rather not have 2.21.0 in a state inconsistent with its makelsr
program.  So if I were to push, I'd need to push the results of a new
makelsr run as well.

Description:
Fix line comments for new snippets

LilyPond line comments should start with "%%" but snippets generated
from Documentation/snippets/new/*.ly previously only used single
percent signs.

Please review this at https://codereview.appspot.com/549820043/

Affected files (+5, -5 lines):
  M scripts/auxiliar/makelsr.py


Index: scripts/auxiliar/makelsr.py
diff --git a/scripts/auxiliar/makelsr.py b/scripts/auxiliar/makelsr.py
index 
3f94a07203a7b8c6391fd24d925bd984a04beed5..73291990b11ecfe4e7c81d6bc8ba43a3eb2c6218
 100755
--- a/scripts/auxiliar/makelsr.py
+++ b/scripts/auxiliar/makelsr.py
@@ -36,12 +36,12 @@ LY_HEADER_LSR = '''%% DO NOT EDIT this file manually; it is 
automatically
 '''
 
 new_lys_marker = "%% generated from %s" % new_lys
-LY_HEADER_NEW = '''%% DO NOT EDIT this file manually; it is automatically
+LY_HEADER_NEW = ''' DO NOT EDIT this file manually; it is automatically
 %s
-%% Make any changes in Documentation/snippets/new/
-%% and then run scripts/auxiliar/makelsr.py
-%%
-%% This file is in the public domain.
+ Make any changes in Documentation/snippets/new/
+ and then run scripts/auxiliar/makelsr.py
+
+ This file is in the public domain.
 ''' % new_lys_marker
 
 options_parser = optparse.OptionParser (

Fix line comments for new snippets (issue 549820043 by d...@gnu.org)

[lemzwerg--- via Discussions on LilyPond development]

LGTM, and please push directly.

https://codereview.appspot.com/549820043/

Re: Problem with LSR import

[David Kastrup]

Valentin Villenave  writes:

> On 4/5/20, David Kastrup  wrote:
>> Valentin Villenave  writes:
>>> It should be available in the next LSR database dump (but there’s at
>>> least one file in the Japanese version that depends on it, so the name
>>> will need to be updated there as well).
>>
>> Thanks!
>
> David, I hope it’s OK if I push that onto staging:
> https://git.savannah.gnu.org/cgit/lilypond.git/commit/?id=e5fcd79862eb74cbe3e2dce651ee9c5ca998506a
> (btw, the html->texinfo conversion doesn’t appear to turn  into
> @itemize bullet as it ideally ought to.)

Patchy didn't appear to mind.  I find it weird that only the Portuguese
.po file seems affected (possibly because all other languages are still
recovering from mostly tracking stable?) and only a Japanese file in a
texidoc directory.

-- 
David Kastrup
My replies have a tendency to cause friction.  To help mitigating
damage, feel free to forward problematic posts to me adding a subject
like "timeout 1d" (for a suggested timeout of 1 day) or "offensive".

Re: Problem with LSR import

[Valentin Villenave]

On 4/5/20, David Kastrup  wrote:
> Valentin Villenave  writes:
>> It should be available in the next LSR database dump (but there’s at
>> least one file in the Japanese version that depends on it, so the name
>> will need to be updated there as well).
>
> Thanks!

David, I hope it’s OK if I push that onto staging:
https://git.savannah.gnu.org/cgit/lilypond.git/commit/?id=e5fcd79862eb74cbe3e2dce651ee9c5ca998506a
(btw, the html->texinfo conversion doesn’t appear to turn  into
@itemize bullet as it ideally ought to.)

V.