gnulib-tool.py: Use absolute imports consistently.
Hi Bruno, I want to make the imports and use of functions from other modules consistent. This patch makes all files use absolute imports like main.py. So this: from pygnulib.GLEmiter import GLEmiter instead of: from .GLEmiter import GLEmiter I want to make another change but would like opinions before writing the patch. Right now we have: import os.path from pygnulib import constants [...] DIRS = constants.DIRS substart = constants.substart isfile = os.path.isfile This works fine, but stops editors from tracking function usage properly. For example, GLEmiter might use 'substart' 30 times but an editor will say there is only 1 usage since we define 'constants.substart' to a variable. Therefore, I would prefer: from constants import substart, bold_escapes, ... If lines get too long you can do something like: from constants import ( substart, bold_escapes, ... ) While making that change it would also be a good time to fix another inconsistency. Right now some files will use a mix of functions prefixed and not prefixed by a module. For example, both 'os.path.isfile' and 'isfile'. I think the best solution is to import our own code directly and use the module prefix for standard library (and third-party code if ever needed). So like this: # Import like this. import os.path from constants import substart # Use them like this. os.path.join(...) substart(...) This feels like what I am used to seeing. It also should be clear when we have situations like 'constants.rmtree()' vs. 'shutil.rmtree()'. Slight exception for the subprocess module, since everyone does 'import subprocess as sp'. It would feel strange not too. :) Any disagreement with those changes? CollinFrom 740b31905915daa3bab5e7cfbd8347ea9f35ef3b Mon Sep 17 00:00:00 2001 From: Collin Funk Date: Sun, 21 Apr 2024 16:58:26 -0700 Subject: [PATCH] gnulib-tool.py: Use absolute imports consistently. * pygnulib/*.py: Update imports. --- ChangeLog | 5 + pygnulib/GLConfig.py| 4 ++-- pygnulib/GLEmiter.py| 12 ++-- pygnulib/GLError.py | 2 +- pygnulib/GLFileSystem.py| 6 +++--- pygnulib/GLImport.py| 18 +- pygnulib/GLInfo.py | 2 +- pygnulib/GLMakefileTable.py | 4 ++-- pygnulib/GLModuleSystem.py | 8 pygnulib/GLTestDir.py | 20 ++-- 10 files changed, 43 insertions(+), 38 deletions(-) diff --git a/ChangeLog b/ChangeLog index b3cef64936..923d59287c 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,8 @@ +2024-04-21 Collin Funk + + gnulib-tool.py: Use absolute imports consistently. + * pygnulib/*.py: Update imports. + 2024-04-21 Collin Funk gnulib-tool.py: Make temporary directories recognizable. diff --git a/pygnulib/GLConfig.py b/pygnulib/GLConfig.py index 05947e8ed3..81973b8e16 100644 --- a/pygnulib/GLConfig.py +++ b/pygnulib/GLConfig.py @@ -22,8 +22,8 @@ import copy import tempfile from typing import Any -from . import constants -from .GLError import GLError +from pygnulib import constants +from pygnulib.GLError import GLError from pygnulib.enums import CopyAction diff --git a/pygnulib/GLEmiter.py b/pygnulib/GLEmiter.py index 91077a0325..bdc0d0649a 100644 --- a/pygnulib/GLEmiter.py +++ b/pygnulib/GLEmiter.py @@ -22,12 +22,12 @@ import re import subprocess as sp from collections.abc import Callable -from . import constants -from .GLInfo import GLInfo -from .GLConfig import GLConfig -from .GLModuleSystem import GLModule -from .GLModuleSystem import GLModuleTable -from .GLMakefileTable import GLMakefileTable +from pygnulib import constants +from pygnulib.GLInfo import GLInfo +from pygnulib.GLConfig import GLConfig +from pygnulib.GLModuleSystem import GLModule +from pygnulib.GLModuleSystem import GLModuleTable +from pygnulib.GLMakefileTable import GLMakefileTable #=== diff --git a/pygnulib/GLError.py b/pygnulib/GLError.py index 47d1e97fdc..1ccbe3b29b 100644 --- a/pygnulib/GLError.py +++ b/pygnulib/GLError.py @@ -19,7 +19,7 @@ # Define global imports #=== import os -from . import constants +from pygnulib import constants #=== diff --git a/pygnulib/GLFileSystem.py b/pygnulib/GLFileSystem.py index 1cad7f5bad..f2a6f64123 100644 --- a/pygnulib/GLFileSystem.py +++ b/pygnulib/GLFileSystem.py @@ -22,10 +22,10 @@ import re import filecmp import subprocess as sp -from . import constants +from pygnulib import constants from pygnulib.enums import CopyAction -from .GLError import GLError -from .GLConfig import GLConfig +from pygnulib.GLError import GLError +from pygnulib.GLConfig import GLConfig #=== diff --git
Re: gnulib-tool.py: Update type hints and docstring.
Collin Funk wrote: > I am still learning about the > type hinting and type checking stuff. Overall I think it is a good > decision though. In the past I remember finding large Python code hard > to follow without them. Absolutely, yes. Bruno
Re: future Python evolution
Hi Paul, > > But the concepts of the shell are stuck in the 40-years-ago past. > > True, but keeping things simple allows for optimizations like PaSH that > can't reasonably be done to Python. > > https://binpa.sh/ Well, I did try PaSh on gnulib-tool — this issue [1] is a testimony. But what can you expect from a parallelization approach? On, say, a quad-core processor you can expect at most a 4x speedup. Which means, the parallelized gnulib-tool.sh would still be 2 times to 25 times slower than the Python rewrite. Also, has PaSh been applied to configure scripts? I recall that configure script parallelization had been a topic for Ralf Wildenhues (before Google swallowed him). Bruno [1] https://github.com/binpash/pash/issues/573
Re: gnulib-tool.py: Update type hints and docstring.
Hi Bruno, On 4/21/24 2:43 PM, Bruno Haible wrote: > Hmm. It's valid code that will work at runtime. I would understand a > "warning", if the tool cannot prove that the code is safe. But "error"? > That's too severe. It's again time for a bug report or two, I would say. Haha, yes maybe "error" is a bit more scary than it should be. It isn't a bug though, just limitations on type checking a dynamic language. I haven't checked but I believe you could use 'dict' which would be equivalent to 'dict[Any, Any]' or even 'dict[str, Any]'. The latter *should* turn off type checking for the value at dict['key'] [1]. There are ways to remove types from the union, called "narrowing", but they wouldn't make much sense when we know a specific dictionary key will have a certain type [2]. Sorry for the not-so great explanations. I am still learning about the type hinting and type checking stuff. Overall I think it is a good decision though. In the past I remember finding large Python code hard to follow without them. > Nah. I think the better cure of the problem is to abandon the 'dict' type > here. That is, create a proper Python class GLFileTable with five attributes. > In this situation, the five attributes are not constrained to have the same > type. I agree. I think that would make things more clear anyways. Maybe I will find an opportunity there to remove the duplicate rewrite_files functions. [1] https://mypy.readthedocs.io/en/stable/dynamic_typing.html#operations-on-any-values [2] https://mypy.readthedocs.io/en/stable/type_narrowing.html#type-narrowing-expressions Collin
Re: GNU gnulib: calling for beta-testers
[CCing bug-gnulib] Mohammad Akhlaghi wrote: > Dear Bruno, > > Thanks for sharing the good news about the speed improvement. Gnuastro > uses Gnulib and it has been very valuable :-). > > I had two questions: > > 1. Will the shell version of Gnulib-tool continue to be the main version > for Gnulib? > > 2. Why did you chose to do this in a high-level and ever-changing > language like Python? If the shell version of gnulib-tool is deprecated, > this will add Python as a bootstrapping dependency of all the projects > that use Gnulib (which is not good, because Python packages usually have > MANY dependencies and can complicate the build environment by > conflicting in their versions with other packages, virtual-env or Conda > also add other complications). > > Cheers, > Mohammad Re 1: This has been discussed just yesterday [1][2]. Re 2: I disagree with the negative connotation in the "ever-changing language" qualification. When the rewrite was started in 2012, we were targeting Python 3.0 to 3.2. Now we are targeting Python 3.7 to 3.12, and we did not have to touch _a single line of code_ due to obsolete or incompatible syntax or features. ISO C, btw, is also "ever-changing", and in practice, it causes more problems than that. (For example, when clang 17 dropped the support for C 89 syntax in favour of C 23.) Why we chose Python? That was a longer thought process. First we had an initial discussion [3][4][5][6], which convinced me that Python is the best choice. Then in 2012 I did a final verification of trends ([7]) that showed that Python's popularity was not likely to go away soon. Regarding bootstrapping dependency: See [8]. Regarding Python packages: gnulib-tool does not use Python packages. It works with a Python compiled from source from the original tarball, with no add-on packages. Bruno [1] https://lists.gnu.org/archive/html/bug-gnulib/2024-04/msg00331.html [2] https://lists.gnu.org/archive/html/bug-gnulib/2024-04/msg00334.html [3] https://lists.gnu.org/archive/html/bug-gnulib/2009-01/msg00034.html [4] https://lists.gnu.org/archive/html/bug-gnulib/2009-01/msg00036.html [5] https://lists.gnu.org/archive/html/bug-gnulib/2009-01/msg00037.html [6] https://lists.gnu.org/archive/html/bug-gnulib/2009-01/msg00059.html [7] http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html [8] https://lists.gnu.org/archive/html/bug-gnulib/2024-04/msg00351.html
Re: gnulib-tool.py: Update type hints and docstring.
Hi Collin,
> > Also, what about the type of filetable? It is described as
> >
> > dict[str, list[str]]
> >
> > but I think it is more
> >
> > dict[str, list[str | tuple[str, str]]]
> >
> > right?
> ...
> Using union types like that is annoying with type checkers though. For
> example, using that type hint and this line of code in
> GLImport.execute():
>
>if [ file
> for file in filetable['all']
> if file.startswith('doc/') ]:
>
> I see this warning under 'startswith' from Pyright:
>
> /home/collin/.local/src/gnulib/pygnulib/GLImport.py:1020:22 - error: Cannot
> access attribute "startswith" for class "tuple[str, str]"
> Attribute "startswith" is unknown (reportAttributeAccessIssue)
>
> or with mypy:
>
> GLImport.py:1020: error: Item "tuple[str, ...]" of "str | tuple[str, str]"
> has no attribute "startswith" [union-attr]
Hmm. It's valid code that will work at runtime. I would understand a
"warning", if the tool cannot prove that the code is safe. But "error"?
That's too severe. It's again time for a bug report or two, I would say.
> It might just be more beneficial to use 'Any' for complex unions like
> that. You could ignore warnings like:
>
>if [ file
> for file in filetable['all']
> if file.startswith('doc/') ]: # type: ignore
>
> but that feels annoying for variables used frequently like that filetable.
Nah. I think the better cure of the problem is to abandon the 'dict' type
here. That is, create a proper Python class GLFileTable with five attributes.
In this situation, the five attributes are not constrained to have the same
type.
Bruno
Re: printf functions without INT_MAX limitation
Paul Eggert wrote: > > 2) Introduce variants of *printf functions, that return a 'ptrdiff_t' > > instead > > of 'int'. (For results longer than PTRDIFF_MAX, they will fail with > > error > > ENOMEM, not EOVERFLOW.) This gives rise to several new gnulib modules. > > This sounds like a good idea. However, shouldn't output-oriented > functions like 'printf' return off_t rather than ptrdiff_t? Hmm. I see how you get there: Because calling fprintf (fp, "%s%s%s%s%s%s%s%s%s%s", s, s, s, s, s, s, s, s, s, s) might return up to nearly 10 * PTRDIFF_MAX. But direct use of off_t has two problems: - off_t is not defined by ISO C; thus it's an odd return type for things like zsprintf. - off_t changes depending on whether gl_LARGEFILE is in use or not; thus it's a bad idea to use it in the API of any shared library or (more generally) any independently-built component. I would prefer to solve this dilemma by defining a new type, say 'zoff_t', that is required to be - at least as wide as off_t, and - at least as wide as ptrditt_t. Similar to how regoff_t is specified in POSIX [1], except that in practice, regoff_t does not satisfy these constraints [2] (because regoff_t is only 32-bits wide - in 32-bit mode on glibc, Solaris, Android, even with -D_FILE_OFFSET_BITS=64, - in 32-bit mode on AIX, even with -D_LARGE_FILES=1, - on Haiku.) In the foreseeable future, zoff_t would be equivalent to int64_t. But it could even be defined as '__int128' with gcc, if need arises. > Also, I'm tempted to use "#define printf zprintf" and leave most of the > source code alone. Perhaps there should be a Gnulib option for that. I see... a pervasive macro, a bit like gl_LARGEFILE. I think we can worry about that when the function-based modules are present. Bruno [1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/regex.h.html [2] https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=tests/test-regex.c;h=175cdf4fe63422fcb1c7de214185caf1b0b4bd24;hb=HEAD#l475
Re: gnulib-tool.py: Update type hints and docstring.
On 4/21/24 4:07 AM, Bruno Haible wrote:
> Also, what about the type of filetable? It is described as
>
> dict[str, list[str]]
>
> but I think it is more
>
> dict[str, list[str | tuple[str, str]]]
>
> right?
Good catch. I think the one you have written is correct because of the
filetable['old'] and filetable['new'] which both have
[(rewritten, original), ...].
Using union types like that is annoying with type checkers though. For
example, using that type hint and this line of code in
GLImport.execute():
if [ file
for file in filetable['all']
if file.startswith('doc/') ]:
I see this warning under 'startswith' from Pyright:
/home/collin/.local/src/gnulib/pygnulib/GLImport.py:1020:22 - error: Cannot
access attribute "startswith" for class "tuple[str, str]"
Attribute "startswith" is unknown (reportAttributeAccessIssue)
or with mypy:
GLImport.py:1020: error: Item "tuple[str, ...]" of "str | tuple[str, str]"
has no attribute "startswith" [union-attr]
It might just be more beneficial to use 'Any' for complex unions like
that. You could ignore warnings like:
if [ file
for file in filetable['all']
if file.startswith('doc/') ]: # type: ignore
but that feels annoying for variables used frequently like that filetable.
Collin
Re: gnulib-tool.py: Don't fail when given a bad module name.
Hi Bruno, On 4/21/24 4:37 AM, Bruno Haible wrote: > The former makes an implicit statement that the function is searching > for the file. The latter does not; it does not say or hint to as what > function internally does. Thus it is better to use the latter wording. Agreed, thanks for the change and advice. My documentation skills are a bit lacking... > It says "Stop." twice but does not actually stop :) Fixed as follows. Nice! Running this passes for me now: $ env GNULIB_TOOL_IMPL=sh+py gnulib-tool --create-testdir --dir aaa abc Collin
Re: printf functions without INT_MAX limitation
On 2024-04-21 09:27, Bruno Haible wrote: 2) Introduce variants of *printf functions, that return a 'ptrdiff_t' instead of 'int'. (For results longer than PTRDIFF_MAX, they will fail with error ENOMEM, not EOVERFLOW.) This gives rise to several new gnulib modules. This sounds like a good idea. However, shouldn't output-oriented functions like 'printf' return off_t rather than ptrdiff_t? Also, I'm tempted to use "#define printf zprintf" and leave most of the source code alone. Perhaps there should be a Gnulib option for that.
printf functions without INT_MAX limitation
Hi Paul,
In several places, we have hit the possibility that the *printf functions
fails with EOVERFLOW (due to a result larger than INT_MAX characters):
- While trying to avoid analyzer warnings from xasprintf(). [1]
- IIRC, you also wrote that in coreutils you would prefer to 'printf'
a variant that doesn't fail if a string happens to be larger than 2 GiB.
I would propose to solve this in three steps:
1) Ensure that vasnprintf() has no INT_MAX limit built-in.
I think, this means doing the %s processing inside of vasnprintf, instead
of using snprintf for this directive. vasnprintf.o becomes a little bit
bigger.
2) Introduce variants of *printf functions, that return a 'ptrdiff_t' instead
of 'int'. (For results longer than PTRDIFF_MAX, they will fail with error
ENOMEM, not EOVERFLOW.) This gives rise to several new gnulib modules.
A sketch of the new module dependencies is as follows ("->" denoting a
module dependency):
c-snprintf -> c-zsnprintf
c-zsnprintf -> c-vasnprintf
c-vasprintf -> c-vazsprintf
c-vazsprintf -> c-vasnprintf
c-vsnprintf -> c-vzsnprintf
c-vzsnprintf -> c-vasnprintf
c-xvasprintf -> c-vazsprintf
dprintf -> dzprintf
dzprintf -> vasnprintf
fprintf -> fzprintf
fzprintf -> vasnprintf
obstack-printf -> obstack-zprintf
obstack-zprintf -> vasnzprintf
printf -> zprintf
zprintf -> vfzprintf
snprintf -> zsnprintf
zsnprintf -> vasnprintf
sprintf -> zsprintf
zsprintf -> vasnprintf
vasnwprintf -> vazsnwprintf
vasprintf -> vazsprintf
vazsprintf -> vasnprintf
vdprintf -> vdzprintf
vdzprintf -> vasnprintf
vfprintf -> vfzprintf
vfzprintf -> vasnprintf
vprintf -> vzprintf
vzprintf -> vfzprintf
vsnprintf -> vzsnprintf
vzsnprintf -> vasnprintf
vsprintf -> vzsprintf
vzsprintf -> vasnprintf
xprintf -> zprintf
xvasprintf -> vazsprintf
and likewise for *printf-posix module variants.
3) Make use of these variants in coreutils etc.
What do you think?
Bruno
PS: Regarding the letter 'z': I originally wanted to use the letter 'l',
but codesearch.debian.net tells me that 'lprintf' is already widely used,
with the meaning of printing a message to a log file. These are the
statistics for use of printf:
a1200
b2300
c1800
d 33400
e 19900
f 1876600
g2400
h 500
i2200
j 5
k1100
l 12300
m 18900
n 900
o4500
p4500
q 700
r2100
s 559100
t 13300
u 800
v 11300
w4700
x8400
y 32
z 300
In order to avoid conflicts with existing code and existing libraries,
a prefix needs to be picked that is little used so far. 'z' seems to fit.
[1] https://lists.gnu.org/archive/html/bug-gnulib/2023-06/msg00014.html
Re: full-source bootstrap and Python
Janneke Nieuwenhuizen wrote: > Are are we creating a problem for > bootstrapping (or even a dependency cycle) when introducing this new > dependency into a certain package. I think you answered this question with "no", when writing in [1]: "Even more recently (2018), the GNU C Library glibc-2.28 adds Python as a build requirement" So, how do you avoid Python when building glibc? Do you use musl libc as a first stage, and only build glibc once a python built with musl exists? Also, from the diagrams in [1][2][3] it looks like the full-source bootstrap uses tarballs frozen in time (make-3.80, gcc-2.95.3, gcc 4.7.3, etc.). So, even if newer versions of 'make' or 'gcc' will use a Python-based gnulib-tool, there won't be a problem, because the bootstrap of these old tarballs will be unaffected. Bruno [1] https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/ [2] https://guix.gnu.org/manual/en/html_node/Reduced-Binary-Seed-Bootstrap.html [3] https://guix.gnu.org/manual/devel/en/html_node/Full_002dSource-Bootstrap.html
Re: beta-tester call draft
Bernhard Voelker writes: > On 4/21/24 01:14, Bruno Haible wrote: >>- install Python on their development machines, > > I'd guess most hosts have python installed nowadays ... the question is > rather which version of it, and how compatible it is: What a host has installed (or could install) is of no consequence. A much more interesting question is: Are are we creating a problem for bootstrapping (or even a dependency cycle) when introducing this new dependency into a certain package. Greetings, Janneke -- Janneke Nieuwenhuizen | GNU LilyPond https://LilyPond.org Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com
Re: future Python evolution
On 4/21/24 16:50, Bruno Haible wrote: Bernhard Voelker wrote: But the concepts of the shell are stuck in the 40-years-ago past. Hehe, Python also had its 33rd birthday already. :-) Good point. Yes, Python occasionally (rarely?) makes incompatible changes. So, I've now created a continuous integration at [2]. If a Python release is made that breaks gnulib-tool, this CI will notify me shortly afterwards, and we will have time to adapt gnulib-tool, even before the new Python release lands in the various distros. great, thanks! Have a nice day, Berny
Re: future Python evolution
On 2024-04-21 07:50, Bruno Haible wrote: But the concepts of the shell are stuck in the 40-years-ago past. True, but keeping things simple allows for optimizations like PaSH that can't reasonably be done to Python. https://binpa.sh/
Re: future Python evolution
Bernhard Voelker wrote: > The shell seems have to been more safe in that regard. But the concepts of the shell are stuck in the 40-years-ago past. That's why it is not recommendable as a programming language for real programs [1]. > I'd guess most hosts have python installed nowadays ... the question is > rather which version of it, and how compatible it is: > now it's <3.7 which is incompatible (according to your mail), > but in future there might come more incompatibilities with newer versions. Good point. Yes, Python occasionally (rarely?) makes incompatible changes. So, I've now created a continuous integration at [2]. If a Python release is made that breaks gnulib-tool, this CI will notify me shortly afterwards, and we will have time to adapt gnulib-tool, even before the new Python release lands in the various distros. Bruno [1] https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00160.html [2] https://gitlab.com/gnulib/gnulib-tool-ci
Re: GNU gnulib: calling for beta-testers
Dear Gnulib developers, Le dimanche 21 avril 2024 à 06:52 -0400, Bruno Haible a écrit : > If you are developer on a package that uses GNU gnulib as part of its > build > system: I have a very simple personal project using gnulib. > 1. Make sure you have Python (version 3.7 or newer) installed on your > machine. > > 2. Update your gnulib checkout. (For some packages, it comes as a git > submodule named 'gnulib'.) > > 3. Set an environment variable that enables checking that the two > implementations behave the same: > > $ export GNULIB_TOOL_IMPL=sh+py > > > 4. Clean the built files of your package > > 5. Regenerate the fetched and generated files of your package. > Depending on > the package, this may be a command such as > > $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR > > If there is a failure, due to differences between the 'sh' and > 'py' > results, please report it to . There are no failures. > The rewritten gnulib-tool was implemented by Dmitry > Selyutin, Collin Funk, and me. You worked well, thank you. Best regards, Vivien
Re: gnulib-tool.py: Make temporary directories recognizable.
Collin Funk wrote: > This patch prefixes temporary directories created by gnulib-tool.py > with "glpy". gnulib-tool.sh uses the 'gl' prefix in func_tmpdir, and I > suppose it is beneficial to differentiate the two. Thanks, applied. Bruno
Re: gnulib-tool.py: Don't fail when given a bad module name.
Hi Collin, > When using --create-testdir with a bad module name: > > $ gnulib-tool.py --create-testdir --dir aaa bad-module-name > gnulib-tool: warning: module bad-module-name doesn't exist > Traceback (most recent call last): > File "/home/collin/.local/src/gnulib/.gnulib-tool.py", line 30, in > main.main_with_exception_handling() > File "/home/collin/.local/src/gnulib/pygnulib/main.py", line 1382, in > main_with_exception_handling > main() > File "/home/collin/.local/src/gnulib/pygnulib/main.py", line 1075, in main > testdir.execute() > File "/home/collin/.local/src/gnulib/pygnulib/GLTestDir.py", line 209, in > execute > requested_licence = requested_module.getLicense() > ^^^ > AttributeError: 'NoneType' object has no attribute 'getLicense' > > This patch removes None from the list of GLModule objects before using > them. Good point. Thanks, patch applied with one tweak: > I also documented the meaning of a None return in > GLModuleSystem.find() since this problem occurred in GLImport > previously. When documenting a function or program, it is sometimes possible to state the same condition with and without referring to the internals of the function / program. It is better to document it without reference to the internals. In this case, one can say if the module description file could not be found or if the module description file does not exist The former makes an implicit statement that the function is searching for the file. The latter does not; it does not say or hint to as what function internally does. Thus it is better to use the latter wording. I also tried the behaviour on gnulib-tool.sh, and got this confusing output: $ ./gnulib-tool.sh --create-testdir --dir=../testdir2 --single-configure foobar signbit ./gnulib-tool.sh: *** file /GNULIB/gnulib-git/./modules/foobar not found ./gnulib-tool.sh: *** Stop. gnulib-tool: warning: module foobar lacks a License ./gnulib-tool.sh: *** file /GNULIB/gnulib-git/./modules/foobar not found ./gnulib-tool.sh: *** Stop. gnulib-tool: warning: module foobar doesn't exist gnulib-tool: warning: module foobar doesn't exist Module list with included dependencies (indented): absolute-header c99 extern-inline ... It says "Stop." twice but does not actually stop :) Fixed as follows. 2024-04-21 Bruno Haible gnulib-tool.sh: In --create-testdir, just warn about a bad module name. * gnulib-tool.sh (func_create_testdir): Validate the modules list. diff --git a/gnulib-tool.sh b/gnulib-tool.sh index 5f0c6f530e..b486e99b1e 100755 --- a/gnulib-tool.sh +++ b/gnulib-tool.sh @@ -6504,6 +6504,9 @@ func_create_testdir () # Except lib-ignore, which leads to link errors when Sun C++ is used. FIXME. modules=`func_all_modules` modules=`for m in $modules; do case $m in config-h | non-recursive-gnulib-prefix-hack | timevar | mountlist | lib-ignore) ;; *) echo $m;; esac; done` + else +# Validate the list of specified modules. +modules=`for module in $modules; do func_verify_module; if test -n "$module"; then echo "$module"; fi; done` fi specified_modules="$modules"
Re: gnulib-tool.py: Update type hints and docstring.
Collin Funk wrote: > Two type hints are incorrectly marked 'dict[str, str]' instead of > 'dict[str, tuple[re.Pattern, str] | None]'. I assume I forgot to > change these when inlining sed expressions to use re.sub(). Thanks, applied. > Now that I think about it, I should have used a list of re.sub() > arguments. But I think these variables are only used by the 'config-h' > module to replace "#if HAVE_CONFIG_H" with "#if 1". Until we need > something more complex than that, how it is currently written should > work fine. Yes, please. Add complexity only when it is needed. Also, what about the type of filetable? It is described as dict[str, list[str]] but I think it is more dict[str, list[str | tuple[str, str]]] right? Bruno
Re: beta-tester call draft
On 4/21/24 01:14, Bruno Haible wrote: - install Python on their development machines, I'd guess most hosts have python installed nowadays ... the question is rather which version of it, and how compatible it is: now it's <3.7 which is incompatible (according to your mail), but in future there might come more incompatibilities with newer versions. The shell seems have to been more safe in that regard. Have a nice day, Berny
GNU gnulib: calling for beta-testers
If you are developer on a package that uses GNU gnulib as part of its build system: gnulib-tool has been known for being slow for many years. We have listened to your complaints. A rewrite of gnulib-tool in another programming language (Python) is ready for beta-testing. It is between 8 times and 100 times faster than the original gnulib-tool. Both implementations should behave identically, that is, produce the same generated files and the same output. You can help us ensure this, through the following steps: 1. Make sure you have Python (version 3.7 or newer) installed on your machine. 2. Update your gnulib checkout. (For some packages, it comes as a git submodule named 'gnulib'.) Like this: $ git checkout master $ git pull Set the environment variable GNULIB_SRCDIR, pointing to this checkout. If the package is using a git submodule named 'gnulib', it is also advisable to do $ git commit -m 'build: Update gnulib submodule to latest.' gnulib (as a preparation for step 5, because the --no-git option does not work as expected in all variants of 'bootstrap'). 3. Set an environment variable that enables checking that the two implementations behave the same: $ export GNULIB_TOOL_IMPL=sh+py 4. Clean the built files of your package: $ make -k distclean 5. Regenerate the fetched and generated files of your package. Depending on the package, this may be a command such as $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR or $ export GNULIB_SRCDIR; ./autopull.sh; ./autogen.sh or, if no such script is available: $ $GNULIB_SRCDIR/gnulib-tool --update If there is a failure, due to differences between the 'sh' and 'py' results, please report it to . 6. If this invocation was successful, you can trust the rewritten gnulib-tool and use it from now on, by setting the environment variable $ export GNULIB_TOOL_IMPL=py 7. Continue with $ ./configure $ make as usual. And enjoy the speed! The rewritten gnulib-tool was implemented by Dmitry Selyutin, Collin Funk, and me. ___ Message sent via Savannah https://savannah.gnu.org/
gnulib-tool.py: Make temporary directories recognizable.
This patch prefixes temporary directories created by gnulib-tool.py with "glpy". gnulib-tool.sh uses the 'gl' prefix in func_tmpdir, and I suppose it is beneficial to differentiate the two. I probably should have done this ~2 months ago when gnulib-tool.py crashed frequently. After testing my /tmp would be full of directories named with the template "tmp", which I assume gets used by other programs too. Thankfully that is less of an issue now since gnulib-tool.py crashes less... CollinFrom 68e59ded4b879011d2a8b3ff3b3342225dcdf312 Mon Sep 17 00:00:00 2001 From: Collin Funk Date: Sun, 21 Apr 2024 00:29:50 -0700 Subject: [PATCH] gnulib-tool.py: Make temporary directories recognizable. * pygnulib/GLConfig.py (GLConfig.__init__): Pass the 'glpy' prefix to mkdtemp. --- ChangeLog| 6 ++ pygnulib/GLConfig.py | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/ChangeLog b/ChangeLog index 6523c89b87..502cdeeaeb 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,9 @@ +2024-04-21 Collin Funk + + gnulib-tool.py: Make temporary directories recognizable. + * pygnulib/GLConfig.py (GLConfig.__init__): Pass the 'glpy' prefix to + mkdtemp. + 2024-04-20 Collin Funk gnulib-tool.py: Don't fail when given a bad module name. diff --git a/pygnulib/GLConfig.py b/pygnulib/GLConfig.py index 19611b823c..05947e8ed3 100644 --- a/pygnulib/GLConfig.py +++ b/pygnulib/GLConfig.py @@ -87,7 +87,7 @@ def __init__(self, errors: bool | None = None) -> None: '''Create new GLConfig instance.''' self.table = dict() -self.table['tempdir'] = tempfile.mkdtemp() +self.table['tempdir'] = tempfile.mkdtemp(prefix='glpy') # Check and store the attributes. # Remove trailing slashes from the directory names. This is necessary # for m4base (to avoid an error in func_import) and optional for the -- 2.44.0
