Re: gnulib-tool: Use the Python implementation by default

[Collin Funk]

Hi Dmitry,

On 4/29/24 11:57 AM, Dmitry Selyutin wrote:
> I've been tracking your progress for a while, even though sporadically and
> remaining silent. I'd like to say "thank you" to Bruno and Collin, who made
> it this far and never surrendered. :-)

Thank you for the kind words! I'm glad you are happy with the changes
and don't mind that I showed up and started messing with your code. :)

> Truth to be told, the code I implemented leaves much to be desired,
> and you were extremely passionate and professional about its quality
> (frankly, it deserved a worse reaction; I apologize for the quality
> and overall style). :-)

No need to apologize! I hope any criticism that I had didn't come off
harsh. If someone told me to rewrite 'gnulib-tool' from scratch the
result of my work would be *far* worse than yours. Thank you for doing
the hard work for me! :D

Collin

Re: gnulib-tool: Use the Python implementation by default

[Bruno Haible]

Hi Dmitry,

The biggest "thank you" is yours, since you did the majority of the work
(my estimations: you 4 months, Collin 2 months, me 1 month).

OO is hard. I was and am still impressed about how your dissection of the
code into classes (GLConfig, GLModule, GLModuleSystem, etc.) stood the test
of time. Collin and I did not have to move code around between classes or
create new concepts that were not present.

Bruno

Re: gnulib-tool: Use the Python implementation by default

[Dmitry Selyutin]

Hi folks, sorry for the long silence!

I've been tracking your progress for a while, even though sporadically and
remaining silent. I'd like to say "thank you" to Bruno and Collin, who made
it this far and never surrendered. :-) Truth to be told, the code I
implemented leaves much to be desired, and you were extremely passionate
and professional about its quality (frankly, it deserved a worse reaction;
I apologize for the quality and overall style). :-) Many thanks to you for
making this come true!

Re: gnulib-tool: Use the Python implementation by default

[Paul Eggert]

On 2024-04-27 11:49, Bruno Haible wrote:

   - In many recent systems, 'python3' exists and 'python' does not.
 This includes Ubuntu 23.10 and 24.04. (No idea why it's different
 on your machine.)


Oh, I see I have the python-is-python3 package installed on my Ubuntu 
platform. I had forgotten that Debian wants users by default to live in 
a compatibility hell where the 'python' command does not work.



I have not seen a single system where 'python' exists, is version 3, and
'python3' does not exist.


I was thinking more of the future, where I expect such systems to exist. 
I suppose we can wait until then.

Re: gnulib-tool: Use the Python implementation by default

[Collin Funk]

On 4/27/24 11:49 AM, Bruno Haible wrote:
>   - In many older systems, 'python3' exists and 'python' exists as well and is
> a 2.7.x version.
> 
> I have not seen a single system where 'python' exists, is version 3, and
> 'python3' does not exist.

There is this PEP which came about with the Python 2 to Python 3
transition:

https://peps.python.org/pep-0394/

If I am interpreting it correctly 'python3' would be correct.

The only other python implementation that I tried was mypy3 which
works fine. I wanted to try Jython but it only supports version 2.7.

Collin

Re: gnulib-tool: Use the Python implementation by default

[Bruno Haible]

Paul Eggert wrote:
> > Are there alternative Python implementations, which could be installed under
> > the name 'python3'?
> 
> On the platforms I currently have terminals open on (Fedora 40, Ubuntu 
> 23.10) it's called 'python'. You can also use 'python3' but I would 
> suggest trying 'python' first if you aren't already doing that. These 
> platforms do not have Python 2, which is typical nowadays.

We would gain nothing by trying 'python' before 'python3'.

  - In many recent systems, 'python3' exists and 'python' does not.
This includes Ubuntu 23.10 and 24.04. (No idea why it's different
on your machine.)

  - In some recent systems, 'python3' exists and 'python' is a symlink to it.
Such as on Fedora 40.

  - In many older systems, 'python3' exists and 'python' exists as well and is
a 2.7.x version.

I have not seen a single system where 'python' exists, is version 3, and
'python3' does not exist.

Bruno

Re: gnulib-tool: Use the Python implementation by default

[Bruno Haible]

Hi Pádraig,

Thanks for the details.

> So this is a general question, unrelated to my particular suggestion.

It _is_ related :-)

> Possible drop in replacements for the python3 command might be ...
> 
> ... pypy, which separates its implementation version from python language 
> support version:
> $ dnf install pypy3
> $ pypy3
> 
>   sys.pypy_version_info
> sys.pypy_version_info(major=7, minor=3, micro=15, releaselevel='final', 
> serial=0)
>   sys.version_info
> sys.version_info(major=3, minor=10, micro=13, releaselevel='final', serial=0)
> 
> ... cinder, which is very close to cpython ...
> $ dnf install docker
> $ sudo systemctl start docker
> $ sudo setfacl --modify user:$USER:rw /var/run/docker.sock
> $ docker run -it --rm ghcr.io/facebookincubator/cinder-runtime:cinder-3.10
> 
> Python 3.10.5+cinder (main, Apr 26 2024, 22:00:58) [GCC 10.3.1 20210422 (Red 
> Hat 10.3.1-1)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> import sys
>  >>> sys.version_info
> sys.version_info(major=3, minor=10, micro=5, releaselevel='final', serial=0)
> 
> 
> There are no other real contenders for drop in python 3 replacement.

OK. This means that people might be tempted to create a symlink
python3 -> pypy3
or
python3 -> cinder.

With the values that you have shown, it will be indeed better to use
sys.version_info.

> Testing shows that a subshell does avoid that issue, so the following works:
> 
>if (python3 -c 'import sys; sys.exit(not sys.version_info >= (3,7))') 
> 2>/dev/null; then

Thanks, I'm taking this one.


2024-04-27  Bruno Haible  

gnulib-tool: Simplify the Python version test.
Suggested by Pádraig Brady in
.
* gnulib-tool: Use Python's sys.version_info, not --version.
* gnulib-tool.py: Use the same version test here. Use --version only to
produce a specific error message.

diff --git a/gnulib-tool b/gnulib-tool
index 441958ae7e..56c4473318 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -143,12 +143,7 @@ case "$GNULIB_TOOL_IMPL" in
   '')
 # Use the Python implementation if a suitable Python version is found
 # in $PATH. This is the same Python version test as in gnulib-tool.py.
-if (python3 --version) >/dev/null 2>/dev/null \
-   && case `python3 --version 2>&1` in
-Python\ 3.[0-6] | Python\ 3.[0-6].*) false ;;
-Python\ 3.*) true ;;
-*) false ;;
-  esac; then
+if (python3 -c 'import sys; sys.exit(not sys.version_info >= (3,7))') 
2>/dev/null; then
   exec "$gnulib_dir/gnulib-tool.py" "$@"
 else
   echo "gnulib-tool: warning: python3 not found or too old, using the slow 
shell-based implementation" 1>&2
diff --git a/gnulib-tool.py b/gnulib-tool.py
index 81537c272c..52389dcd78 100755
--- a/gnulib-tool.py
+++ b/gnulib-tool.py
@@ -131,17 +131,21 @@
 func_gnulib_dir
 
 # Check the Python version.
-if (python3 --version) >/dev/null 2>/dev/null; then
-  case `python3 --version 2>&1` in
-Python\ 3.[0-6] | Python\ 3.[0-6].*)
-  func_fatal_error "python3 is too old (minimum required version is 3.7); 
try setting GNULIB_TOOL_IMPL=sh" ;;
-Python\ 3.*)
-  ;;
-*)
-  func_fatal_error "python3 version is unsupported" ;;
-  esac
+if (python3 -c 'import sys; sys.exit(not sys.version_info >= (3,7))') 
2>/dev/null; then
+  :
 else
-  func_fatal_error "python3 not found; try setting GNULIB_TOOL_IMPL=sh"
+  if (python3 --version) >/dev/null 2>/dev/null; then
+case `python3 --version 2>&1` in
+  Python\ 3.[0-6] | Python\ 3.[0-6].*)
+func_fatal_error "python3 is too old (minimum required version is 
3.7); try setting GNULIB_TOOL_IMPL=sh" ;;
+  Python\ 3.*)
+;;
+  *)
+func_fatal_error "python3 version is unsupported" ;;
+esac
+  else
+func_fatal_error "python3 not found; try setting GNULIB_TOOL_IMPL=sh"
+  fi
 fi
 
 # Tell Python to store the compiled bytecode outside the gnulib directory.

Re: gnulib-tool: Use the Python implementation by default

[Paul Eggert]

On 2024-04-27 06:32, Bruno Haible wrote:

Are there alternative Python implementations, which could be installed under
the name 'python3'?


On the platforms I currently have terminals open on (Fedora 40, Ubuntu 
23.10) it's called 'python'. You can also use 'python3' but I would 
suggest trying 'python' first if you aren't already doing that. These 
platforms do not have Python 2, which is typical nowadays.


There are other Python implementations - for example, GraalVM supports 
Python 3.8 and the GraalVM folks say they do a lot of compatibility 
checking. If people want (and have the time) to check portability of the 
new Python code in Gnulib, that might be a good alternative to try. 
Among other things, they claim a big performance improvement over 
CPython - I don't know how well that would work for us though.

Re: gnulib-tool: Use the Python implementation by default

[Pádraig Brady]

On 27/04/2024 14:32, Bruno Haible wrote:

Hi Pádraig,


+if (python3 --version) >/dev/null 2>/dev/null \
+   && case `python3 --version 2>&1` in
+Python\ 3.[0-6] | Python\ 3.[0-6].*) false ;;
+Python\ 3.*) true ;;
+*) false ;;
+  esac; then


It may be preferable to state the supported version directly,


I wanted a short warning message. In most cases the absence of python3
will be the problem. It should be rare than anyone still uses an old
version.


and replace the above 6 lines with:

if python3 -c 'import sys; sys.exit(not sys.version_info >= (3,7))' 
2>/dev/null; then


Thanks for the hint. I admit that I did not think at it. But is it
actually better?


Only that's it's shorter, and more directly states the supported version.
No biggie, just a suggestion.



Are there alternative Python implementations, which could be installed under
the name 'python3'?


So this is a general question, unrelated to my particular suggestion.
I think the version check is fine, and should be quite independent from python 
implementation.
Possible drop in replacements for the python3 command might be ...

... pypy, which separates its implementation version from python language 
support version:
$ dnf install pypy3
$ pypy3

 sys.pypy_version_info
sys.pypy_version_info(major=7, minor=3, micro=15, releaselevel='final', 
serial=0)
 sys.version_info
sys.version_info(major=3, minor=10, micro=13, releaselevel='final', serial=0)

... cinder, which is very close to cpython ...
$ dnf install docker
$ sudo systemctl start docker
$ sudo setfacl --modify user:$USER:rw /var/run/docker.sock
$ docker run -it --rm ghcr.io/facebookincubator/cinder-runtime:cinder-3.10

Python 3.10.5+cinder (main, Apr 26 2024, 22:00:58) [GCC 10.3.1 20210422 (Red 
Hat 10.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version_info
sys.version_info(major=3, minor=10, micro=5, releaselevel='final', serial=0)


There are no other real contenders for drop in python 3 replacement.
For example codon or mojo are not designed as drop in replacements.
I expect any future drop in replacements would follow the model above.


Note also that your snippet would not replace the 6 lines of code, only the
last 5 lines, because with Solaris 10 /bin/sh your snippet produces the
diagnostic
   python3: not found


Out of interest, I dug out an old dsa ssh key from a backup, to get access to a 
solaris 10 machine.
Testing shows that a subshell does avoid that issue, so the following works:

  if (python3 -c 'import sys; sys.exit(not sys.version_info >= (3,7))') 
2>/dev/null; then

cheers,
Pádraig

Re: gnulib-tool: Use the Python implementation by default

[Bruno Haible]

Hi Pádraig,

> > +if (python3 --version) >/dev/null 2>/dev/null \
> > +   && case `python3 --version 2>&1` in
> > +Python\ 3.[0-6] | Python\ 3.[0-6].*) false ;;
> > +Python\ 3.*) true ;;
> > +*) false ;;
> > +  esac; then
> 
> It may be preferable to state the supported version directly,

I wanted a short warning message. In most cases the absence of python3
will be the problem. It should be rare than anyone still uses an old
version.

> and replace the above 6 lines with:
> 
>if python3 -c 'import sys; sys.exit(not sys.version_info >= (3,7))' 
> 2>/dev/null; then

Thanks for the hint. I admit that I did not think at it. But is it
actually better?

Are there alternative Python implementations, which could be installed under
the name 'python3'?

Note also that your snippet would not replace the 6 lines of code, only the
last 5 lines, because with Solaris 10 /bin/sh your snippet produces the
diagnostic
  python3: not found

Bruno

Re: gnulib-tool: Use the Python implementation by default

[Pádraig Brady]

On 26/04/2024 10:48, Bruno Haible wrote:

I committed this patch, which activates the Python rewrite of gnulib-tool
for all users who have Python installed, without the need to set any
environment variable. I'll make announcements to info-gnu and planet.gnu.org
soon.



+if (python3 --version) >/dev/null 2>/dev/null \
+   && case `python3 --version 2>&1` in
+Python\ 3.[0-6] | Python\ 3.[0-6].*) false ;;
+Python\ 3.*) true ;;
+*) false ;;
+  esac; then


It may be preferable to state the supported version directly,
and replace the above 6 lines with:

  if python3 -c 'import sys; sys.exit(not sys.version_info >= (3,7))' 
2>/dev/null; then


+  exec "$gnulib_dir/gnulib-tool.py" "$@"
+else
+  echo "gnulib-tool: warning: python3 not found or too old, using the slow shell-based 
implementation" 1>&2
+  exec "$gnulib_dir/gnulib-tool.sh" "$@"
+fi
+;;


I've updated to the latest gnulib now in coreutils,
and the auto python selection works well.

thanks!
Pádraig

Re: gnulib-tool: Use the Python implementation by default

[Bruno Haible]

Hi Collin,

> Should Python 3.7 be added to DEPENDENCIES or
> should we hold off on that for a bit? Since gnulib-tool.sh can still
> be used.

Good point. Done like this:


2024-04-26  Bruno Haible  

Document prerequisite for gnulib-tool.
Suggested by Collin Funk in
.
* DEPENDENCIES: Add Python 3.7 as "recommended".

diff --git a/DEPENDENCIES b/DEPENDENCIES
index e396adbd80..b700c05bc6 100644
--- a/DEPENDENCIES
+++ b/DEPENDENCIES
@@ -102,6 +102,19 @@ at any time.
   + Download:
 https://ftp.gnu.org/gnu/gawk/
 
+* Python 3.7 or newer.
+  * Recommended.
+Without it, gnulib-tool is between 8 times and 100 times slower and may 
lack
+functionality added after April 2024.
+  + Homepage:
+https://www.python.org/
+  + Download:
+https://www.python.org/downloads/
+  + Pre-built package name:
+- On Debian and Debian-based systems: python3,
+- On Red Hat distributions: python3.
+- Other: https://repology.org/project/python/versions
+
 * GNU gettext.
   + If your project wants to follow GNU Coding Standards:
 + Always use the newest available gettext release, see

Re: gnulib-tool: Use the Python implementation by default

[Collin Funk]

Hi Bruno,

On 4/26/24 2:48 AM, Bruno Haible wrote:
> I committed this patch, which activates the Python rewrite of gnulib-tool
> for all users who have Python installed, without the need to set any
> environment variable. I'll make announcements to info-gnu and planet.gnu.org
> soon.

Looks good, thanks. Should Python 3.7 be added to DEPENDENCIES or
should we hold off on that for a bit? Since gnulib-tool.sh can still
be used.

> Also I couldn't resist pushing a tag named 'v1.0'. We don't make releases,
> but it's an indication that Gnulib is mature now.

For what it's worth I thought it was already mature, but the v0.1 to
v1.0 jump makes my work feel more important so I'm happy with it. :)

> Many thanks to Dmitry Selyutin for the ground work in 2012, and to Collin Funk
> for the finishing work in the last two months!

Many thanks to Dmitry to doing most of the hard work and to Bruno for
checking my patches and helping me understand the shell script.

Collin

gnulib-tool: Use the Python implementation by default

[Bruno Haible]

I committed this patch, which activates the Python rewrite of gnulib-tool
for all users who have Python installed, without the need to set any
environment variable. I'll make announcements to info-gnu and planet.gnu.org
soon.

Also I couldn't resist pushing a tag named 'v1.0'. We don't make releases,
but it's an indication that Gnulib is mature now.

Many thanks to Dmitry Selyutin for the ground work in 2012, and to Collin Funk
for the finishing work in the last two months!


2024-04-26  Bruno Haible  

gnulib-tool: Use the Python implementation by default.
* gnulib-tool: If GNULIB_TOOL_IMPL is not set: Use the Python
implementation if a suitable 'python3' program is found in $PATH;
otherwise print a warning and use the shell implementation.

diff --git a/gnulib-tool b/gnulib-tool
index 85b62883c6..441958ae7e 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -140,7 +140,22 @@ func_gnulib_dir ()
 func_gnulib_dir
 
 case "$GNULIB_TOOL_IMPL" in
-  '' | sh)
+  '')
+# Use the Python implementation if a suitable Python version is found
+# in $PATH. This is the same Python version test as in gnulib-tool.py.
+if (python3 --version) >/dev/null 2>/dev/null \
+   && case `python3 --version 2>&1` in
+Python\ 3.[0-6] | Python\ 3.[0-6].*) false ;;
+Python\ 3.*) true ;;
+*) false ;;
+  esac; then
+  exec "$gnulib_dir/gnulib-tool.py" "$@"
+else
+  echo "gnulib-tool: warning: python3 not found or too old, using the slow 
shell-based implementation" 1>&2
+  exec "$gnulib_dir/gnulib-tool.sh" "$@"
+fi
+;;
+  sh)
 exec "$gnulib_dir/gnulib-tool.sh" "$@" ;;
   py)
 exec "$gnulib_dir/gnulib-tool.py" "$@" ;;