Re: debugging gnulib-tool.py

[Collin Funk]

On 3/15/24 5:27 PM, Bruno Haible wrote:
> * Choice: It seems that PyCharm, VSCode, Spyder, Eclipse with PyDev are the
>   most prominent candidates.

I used PyCharm for a course I took in college since the professor
wanted us to. I remember liking it. It seems that the source of the
PyCharm Community Edition is available under the Apache 2.0 license;
see the python subdirectory [1].

They may distribute builds with proprietary software like Microsoft
does with VSCode though. I do not use it anymore so I am not sure.

>   So, I chose Eclipse with PyDev.
> 
> Eclipse has a long learning curve (2 weeks from the ground), therefore here
> is how I configure it.

Thanks for the guide. I remember using Eclipse a few years ago and
being a bit confused by all the buttons everywhere. This should help.

I was looking for debugging programs a few days ago before I got tired
and fell asleep without trying any...

Have you tried pdb by any chance [2]? It looks like it might be
someone comfy for gdb users. Maybe I am setting my hopes too high
though.

[1] https://github.com/JetBrains/intellij-community/tree/master
[2] https://docs.python.org/3/library/pdb.html

Collin

Re: [PATCH] gnulib-tool.py: Follow gnulib-tool changes, part 58.

[Collin Funk]

Hi Bruno,

On 3/15/24 6:09 PM, Bruno Haible wrote:
> To make these two options interfere with each other, so that the last one
> wins, how about using the same variable for both?

Well, now I feel silly...
Let me know how this looks. I've imported CopyAction to main and use
that in a similar way to what you wrote.

I've used the name copymode instead of copyaction to match
gnulib-tool.sh. This also means changing GLConfig to store the Enum
itself instead of booleans for symlink and hardlink.

> * GLConfig.py:
> 
> self.resetLHardlink()
> if lhardlink != None:
> self.resetLHardlink()

Yes, this was incorrect.

> * GLFileSystem.py:
> 
> constants.hardlink(lookedup, joinpath(destdir, rewritten))
> 
>   Please import the 'hardlink' symbol, so that you can reference it like this:
> 
> hardlink(lookedup, joinpath(destdir, rewritten))

I didn't do this since the invocations of link_if_changed and substart
used the 'constants.' prefix. Since you wanted this one change I have
gone ahead and done the rest of them.

In main(), all of the classes are prefixed with 'classes.'. Because
there are many occurrences of it, I have stuck to that convention for
now.

In GLConfig.py, I have also used the 'classes.' for the CopyAction
Enum. I tried to get rid of this but I ran into recursive import
errors.

I rather submit another patch later fixing that. The import stuff is a
bit confusing to me and makes me miss #include very much. I rather not
make this patch harder to read.

> * GLTestDir.py line 369:
> 
> if self.filesystem.shouldLink(src, lookedup) == 
> CopyAction.Hardlink:
> 
>   Are you sure this shouldn't be an 'elif'?

It should be 'elif', good catch. In the shell script it was an
'if $condition' that was nested under an 'else'. My eyes skipped over
the else so I misinterpreted it. Thanks.

>   According to the TODO file, the patch should implement the options
> -S | --more-symlinks
> -H | --more-hardlinks
>   But it doesn't do so.

These are just aliases for the other options correct?

I've added them to the corresponding add_argument call like this:

 parser.add_argument('-h', '-H', '--hardlink', '--more-hardlinks',
 ...)  # rest

And similarly for --symlink.

CollinFrom 1568aff94b28db067de7d83ccb4630de2e1c4227 Mon Sep 17 00:00:00 2001
From: Collin Funk 
Date: Fri, 15 Mar 2024 19:58:27 -0700
Subject: [PATCH] gnulib-tool.py: Follow gnulib-tool changes, part 58.

Follow gnulib-tool change
2017-05-21  Bruno Haible  
gnulib-tool: Add options to create hard links.

* pygnulib/GLConfig.py (GLConfig.__init__): Add 'copymode' and
'lcopymode' to the parameter list. Initialize them.
(GLConfig.default): Don't use symbolic or hard links by default.
(GLConfig.checkCopyMode, GLConfig.setCopyMode, GLConfig.resetCopyMode):
New functions to modify and check the method for copying non --local-dir
files.
(GLConfig.checkLCopyMode, GLConfig.setLCopyMode)
(GLConfig.resetLCopyMode): New functions to modify and check the method
for copying --local-dir files.
(GLConfig.checkSymbolic, GLConfig.resetSymbolic, GLConfig.setSymbolic)
(GLConfig.checkLSymbolic, GLConfig.resetLSymbolic)
(GLConfig.setLSymbolic): Remove unused functions. The functionality of
these are now implemented in the *CopyMode() and *LCopyMode() variants.
* pygnulib/GLFileSystem.py (CopyAction.Hardlink): New Enum value to
describe hard links.
(GLFileSystem.shouldLink): Check if hard links should be used.
(GLFileAssistant.add, GLFileAssistant.update): Try to hard link if
enabled. Copy the file if linking fails.
(GLFileAssistant.add_or_update): Remove temporary files unconditionally.
* pygnulib/GLInfo.py (GLInfo.usage): Document new options in the usage
message.
* pygnulib/GLTestDir.py (GLTestDir.execute): Try to hard link if
enabled. Copy the file if linking fails.
* pygnulib/classes.py: Importy the CopyAction Enum.
* pygnulib/constants.py (hardlink): New function based on
symlink_relative.
* pygnulib/main.py (main): Add new options --hardlink and
--local-hardlink. Invoke 'git update-index --refresh' to mitigate the
effects of the hard links on git.
---
 ChangeLog| 37 
 gnulib-tool.py.TODO  | 31 +-
 pygnulib/GLConfig.py | 93 +---
 pygnulib/GLFileSystem.py | 63 +--
 pygnulib/GLInfo.py   |  4 ++
 pygnulib/GLTestDir.py|  4 +-
 pygnulib/classes.py  |  4 +-
 pygnulib/constants.py| 20 +
 pygnulib/main.py | 41 ++
 9 files changed, 190 insertions(+), 107 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 0822e36cc4..9997860fc2 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,40 @@
+2024-03-15  Collin Funk  
+
+
+	gnulib-tool.py: Follow gnulib-tool changes, part 58.
+	Follow gnulib-tool change
+	2017-05-21  Bruno Haible  
+	gnulib-tool: Add options to create hard links.
+	* pygnulib/GLConfig.py 

Re: why is gnulib-tool.sh slow

[Bruno Haible]

Collin Funk wrote:
> > Wow it is very much faster!  \o/
> 
> That result is interesting. I wonder what part takes the shell script
> so long.

Simon's oath-toolkit package invokes gnulib-tool 5 times. So, the average
run time of each gnulib-tool.sh invocation was 9 sec. Which is not
terribly long.

It looks like each of the steps (computing the set of modules, computing
the set of files, actually copying the files) take a couple of seconds.

IMO, the main problem with sh or bash as a scripting language is that
it lacks in-process string processing facilities. Simply dissecting
a string like
   'foo[test $HAVE_FOO = 0]'
into 'foo' and 'test $HAVE_FOO = 0' takes one or more pipes and 'sed'
invocations. Even if 'sed', like 'echo', was a shell built-in, there
would still be the overhead of a fork() and a pipe() system call.

This way of doing computations — delegate computations to
specialized programs — is influenced by
  (1) the "Unix philosophy" [1],
  (2) composition through program invocation.

(2) has nowadays mostly been replaced by composition through libraries.
(2) is an artifact of the small memory sizes (max. 64 KB) that a program
could use up to ca. 1987. Even on a computer with 1 MB of RAM, you
could not map 25 libraries, each of 10 KB size, into the address space
of the same process. Thus, they used program invocation as a way of
building larger software from small pieces.

Nowadays, composition through libraries is the predominant approach
for building larger software. Thus, most programming environments and
languages contain string manipulation libraries, numerical libraries,
networking libraries, etc. all available in the same process. But
the sh / bash language still doesn't: it still uses subprocesses and
pipes for even the most basic things.

Bruno

[1] https://en.wikipedia.org/wiki/Unix_philosophy

Re: gnulib-tool: Obey environment variable GNULIB_TOOL_IMPL

[Bruno Haible]

Simon Josefsson wrote:
> OATH Toolkit fails with sh+py though...  isn't the --local-dir part
> working?  It doesn't notice the patches in liboath/gl/override.

Indeed. It reads the gl_LOCAL_DIR value from gnulib-cache.m4 but
then overwrites it with []. This patch fixes it.


2024-03-15  Bruno Haible  

gnulib-tool.py: Don't lose the gl_LOCAL_DIR value from gnulib-cache.m4.
Reported by Simon Josefsson in
.
* pygnulib/GLConfig.py (GLConfig.default): For the 'localpath' property,
return [], not ''.

diff --git a/pygnulib/GLConfig.py b/pygnulib/GLConfig.py
index fea65803d9..ce34de8aec 100644
--- a/pygnulib/GLConfig.py
+++ b/pygnulib/GLConfig.py
@@ -282,7 +282,8 @@ class GLConfig(object):
 return 2.64
 elif key == 'verbosity':
 return 0
-elif key in ['modules', 'avoids', 'tests', 'incl_test_categories', 
'excl_test_categories']:
+elif key in ['localpath', 'modules', 'avoids', 'tests',
+ 'incl_test_categories', 'excl_test_categories']:
 return list()
 elif key in ['libtool', 'lgpl', 'gnu_make', 'automake_subdir', 
'conddeps', 'symbolic',
  'lsymbolic', 'libtests', 'dryrun']:

Re: [PATCH] gnulib-tool.py: Follow gnulib-tool changes, part 58.

[Bruno Haible]

Hi Collin,

> This patch implements --hardlink and --local-hardlink.

Thanks; that's major (and tricky), indeed.

> Notice the arguments '-s --hardlink --symlink -h'. I tried to make
> sure gnulib-tool.py behaved similarly to gnulib-tool.sh in this case.
> 
> I beleive the correct behavior is to use the last option given so
> '-h'. ...

Yes.

> It doesn't seem there is a good way to do this with Python's argparse,
> though I may not have looked at the documentation hard enough [1].

Well, when the code contains

parser.add_argument('-s', '--symbolic', '--symlink',
dest='symlink',
default=None,
action='store_true')
...
parser.add_argument('-h', '--hardlink',
dest='hardlink',
default=None,
action='store_true')

each of -s and -h will store into a different variable. So, they will
act independently.

To make these two options interfere with each other, so that the last one
wins, how about using the same variable for both?

class CopyAction(Enum):
Copy = 0
Symlink = 1
Hardlink = 2

parser.add_argument('-s', '--symbolic', '--symlink',
dest='copyaction',
default=None,
action='store_const', const=CopyAction.Symlink)
...
parser.add_argument('-h', '--hardlink',
dest='copyaction',
default=None,
action='store_const', const=CopyAction.Hardlink)

> I ended up doing this which seems to work, though is not very pretty:
> 
> # --symlink and --hardlink are mutually exclusive.
> # Use the last one given.
> if symlink and hardlink:
> optindex_table = {}
> options = list(reversed(sys.argv[1:]))
> options_count = len(options)
> for option in ['-s', '--symbolic', '--symlink', '-h', '--hardlink']:
> if option in options:
> optindex_table[option] = options_count - options.index(option)
> last_option = max(optindex_table, key=optindex_table.get)
> # Disable the option that is not the last one.
> if last_option in ['-s', '--symbolic', '--symlink']:
> hardlink = False
> else:  # last_option is --hardlink or equivalent.
> symlink = False
> 
> And something similar for the --local-* variants.

Yes, this is not pretty. I wish you can remove this code, if the outline
above works.

Other than that:

* GLConfig.py:

self.resetLHardlink()
if lhardlink != None:
self.resetLHardlink()

  The last of these 3 lines looks wrong.

* GLFileSystem.py:

constants.hardlink(lookedup, joinpath(destdir, rewritten))

  Please import the 'hardlink' symbol, so that you can reference it like this:

hardlink(lookedup, joinpath(destdir, rewritten))

* GLTestDir.py line 369:

if self.filesystem.shouldLink(src, lookedup) == 
CopyAction.Hardlink:

  Are you sure this shouldn't be an 'elif'?

* main.py:

  According to the TODO file, the patch should implement the options
-S | --more-symlinks
-H | --more-hardlinks
  But it doesn't do so.

Bruno

debugging gnulib-tool.py

[Bruno Haible]

Hi,

In order to debug Simon's report
,
I decided to use a GUI debugger, rather than the primitive print(...)-based
debugging.

* Choice: It seems that PyCharm, VSCode, Spyder, Eclipse with PyDev are the
  most prominent candidates.

  VSCode is "modern", but
- the builds by Microsoft contain proprietary software,
- you have to explicitly opt-out from telemetry collection if you want
  your privacy to be preserved,
- last time I used it (two years ago), it had bugs related to caches,
  with the effect that syntax errors were shown that I already had
  corrected.

  So, I chose Eclipse with PyDev.

Eclipse has a long learning curve (2 weeks from the ground), therefore here
is how I configure it.

* Download and configuration:
  - Eclipse IDE from https://www.eclipse.org/downloads/packages/
(either the build for Java or for C/C++ should work fine).
  - PyDev from https://www.pydev.org/download.html
section "Install as Plugin".
(Don't use LiClipse, since the license costs money.)
  - Follow https://www.pydev.org/manual_101_install.html,
replacing http://www.pydev.org/updates
with  https://www.pydev.org/updates
  - Once installed, restart the IDE.
  - Window > Preferences > PyDev > Interpreters > Python Interpreter:
New > path: /usr/bin/python3

* Create a project:
  Let GNULIB_DIR be my gnulib checkout.
  - File > New > Project... > PyDev > PyDev Project. Call it 'gnulib'.
Note that this is a project inside the Eclipse workspace, so far
unrelated to the GNULIB_DIR.
  - Popup menu > New > Link to Existing Source
Name: pygnulib
Path: /pygnulib

* Create a run configuration:
  - Run > Run Configurations... > Python Run
Popup menu > New configuration
  Name: gnulib-tool.py
  Main > Project: gnulib
  Environment > Add
Variable: PYTHONPATH
Value: 
(This is the same hack as in /gnulib-tool.py.)
  Arguments > Program arguments: --help
  - Test it: Run this configuration.

* Create a debug configuration:
  - Run > Debug Configurations... > gnulib-tool.py
Popup menu > Duplicate
  - In the duplicate, set
Arguments > Working directory: /tmp/oath-toolkit/lib
Arguments > Program arguments: --add-import

* Debug it:
  - Open GLImport.py.
  - On the left-hand border of this editor, do "Add breakpoint".
  - Run > Debug Configurations... > pick the duplicate. Press Debug.

Re: planning for beta-testing gnulib-tool.py

[Bruno Haible]

Collin Funk wrote:
> I'm still finishing some things before we can recommend it for
> testing.

Right. It would not be productive if we had different package
maintainers report the same bug(s) at the same time.

> But Wget seems like a good project for testing. Do you have
> any thoughts Bruno?

Among the packages listed in users.txt, I can imagine that the
following — whose maintainers have occasionally shown up on this
mailing list — would respond to beta-testing requests in a
friendly manner:

  autobuild   https://git.savannah.nongnu.org/gitweb/?p=autobuild.git;a=tree
  bison   https://git.savannah.gnu.org/gitweb/?p=bison.git;a=tree
  clisp   https://gitlab.com/gnu-clisp/clisp
  coreutils   https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=tree
  diffutils   https://git.savannah.gnu.org/gitweb/?p=diffutils.git;a=tree
  emacs   https://git.savannah.gnu.org/gitweb/?p=emacs.git;a=tree
  findutils   https://git.savannah.gnu.org/gitweb/?p=findutils.git;a=tree
  gettext https://git.savannah.gnu.org/gitweb/?p=gettext.git;a=tree
  gnutls  https://www.gnutls.org/
  https://gitlab.com/gnutls/gnutls.git
  gsasl   https://git.savannah.gnu.org/gitweb/?p=gsasl.git;a=tree
  grephttps://git.savannah.gnu.org/gitweb/?p=grep.git;a=tree
  gziphttps://git.savannah.gnu.org/gitweb/?p=gzip.git;a=tree
  libffcall   https://git.savannah.gnu.org/gitweb/?p=libffcall.git;a=tree
  libiconvhttps://git.savannah.gnu.org/gitweb/?p=libiconv.git;a=tree
  libidn  https://git.savannah.gnu.org/gitweb/?p=libidn.git;a=tree
  libidn2 https://gitlab.com/libidn/libidn2
  libntlm https://git.savannah.nongnu.org/gitweb/?p=libntlm.git;a=tree
  libunistringhttps://git.savannah.gnu.org/gitweb/?p=libunistring.git;a=tree
  libvirt https://libvirt.org/
  https://libvirt.org/git/?p=libvirt.git;a=tree
  m4  https://git.savannah.gnu.org/gitweb/?p=m4.git;a=tree
  pspphttps://git.savannah.gnu.org/gitweb/?p=pspp.git;a=tree
  wgethttps://git.savannah.gnu.org/gitweb/?p=wget.git;a=tree
  wget2   https://git.savannah.gnu.org/gitweb/?p=wget/wget2.git;a=tree

Bruno

Re: gnulib-tool: Obey environment variable GNULIB_TOOL_IMPL

[Bruno Haible]

Yesterday I did:

>   * gnulib-tool: New file, based on gnulib-tool.sh.

Here's a small improvement.


2024-03-15  Bruno Haible  

gnulib-tool: Enhance last patch.
* gnulib-tool: If gnulib-tool.sh failed but gnulib-tool.py succeeded,
report that and don't erase the outputs and log files.

diff --git a/gnulib-tool b/gnulib-tool
index 9f353550d5..d7b6d33912 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -179,10 +179,14 @@ case "$GNULIB_TOOL_IMPL" in
 "$gnulib_dir/gnulib-tool.sh" "$@" >"$tmp-sh-out" 2>"$tmp-sh-err"
 shrc=$?
 if test $shrc != 0; then
-  cat "$tmp-sh-out"
-  cat "$tmp-sh-err" >&2
-  rm -rf "$tmp" "$tmp-sh-out" "$tmp-sh-err" "$tmp-py-out" "$tmp-py-err"
-  exit $shrc
+  if test $pyrc = 0; then
+func_fatal_error "gnulib-tool.sh failed but gnulib-tool.py 
succeeded! Inspect $tmp-sh-err and $tmp-py-err."
+  else
+cat "$tmp-sh-out"
+cat "$tmp-sh-err" >&2
+rm -rf "$tmp" "$tmp-sh-out" "$tmp-sh-err" "$tmp-py-out" 
"$tmp-py-err"
+exit $shrc
+  fi
 fi
 if test $pyrc != 0; then
   func_fatal_error "gnulib-tool.sh succeeded but gnulib-tool.py 
failed! Inspect $tmp/ and $tmp-py-err."
@@ -218,10 +222,14 @@ case "$GNULIB_TOOL_IMPL" in
 "$gnulib_dir/gnulib-tool.sh" "$@" >"$tmp-sh-out" 2>"$tmp-sh-err"
 shrc=$?
 if test $shrc != 0; then
-  cat "$tmp-sh-out"
-  cat "$tmp-sh-err" >&2
-  rm -rf "$tmp" "$tmp-sh-out" "$tmp-sh-err" "$tmp-py-out" "$tmp-py-err"
-  exit $shrc
+  if test $pyrc = 0; then
+func_fatal_error "gnulib-tool.sh failed but gnulib-tool.py 
succeeded! Inspect $tmp-sh-err and $tmp-py-err."
+  else
+cat "$tmp-sh-out"
+cat "$tmp-sh-err" >&2
+rm -rf "$tmp" "$tmp-sh-out" "$tmp-sh-err" "$tmp-py-out" 
"$tmp-py-err"
+exit $shrc
+  fi
 fi
 if test $pyrc != 0; then
   func_fatal_error "gnulib-tool.sh succeeded but gnulib-tool.py 
failed! Inspect $tmp/ and $tmp-py-err."
@@ -248,10 +256,14 @@ case "$GNULIB_TOOL_IMPL" in
 "$gnulib_dir/gnulib-tool.sh" "$@" >"$tmp-sh-out" 2>"$tmp-sh-err"
 shrc=$?
 if test $shrc != 0; then
-  cat "$tmp-sh-out"
-  cat "$tmp-sh-err" >&2
-  rm -rf "$tmp-sh-out" "$tmp-sh-err" "$tmp-py-out" "$tmp-py-err"
-  exit $shrc
+  if test $pyrc = 0; then
+func_fatal_error "gnulib-tool.sh failed but gnulib-tool.py 
succeeded! Inspect $tmp-sh-err and $tmp-py-err."
+  else
+cat "$tmp-sh-out"
+cat "$tmp-sh-err" >&2
+rm -rf "$tmp-sh-out" "$tmp-sh-err" "$tmp-py-out" "$tmp-py-err"
+exit $shrc
+  fi
 fi
 if test $pyrc != 0; then
   func_fatal_error "gnulib-tool.sh succeeded but gnulib-tool.py 
failed! Inspect $tmp-py-err."

Re: planning for beta-testing gnulib-tool.py

[Collin Funk]

Hi Darshit,

On 3/14/24 12:12 PM, Darshit Shah wrote:
> I'm following the development of gnulib-tool.py pretty closely. Wget and 
> Wget2 have tried to use the Python version multiple times in the past with 
> varying levels of success. 

I am enjoying working on it, so hopefully this time will be more
successful. :)

> I'll volunteer my time for using GNU Wget and Wget2 as initial beta testers 
> of the new Python based gnulib-tool. 

I am not too familiar with Wget but since I already have Wget2 cloned
I attempted to use gnulib-tool.py to run the bootstrap script:

$ env GNULIB_TOOL_IMPL=py ./bootstrap
$ ./configure
$ make all
$ make check

This builds successfully and passes the test cases. However, as
expected the output of gnulib-tool.py does not match gnulib-tool.sh.

I'm still finishing some things before we can recommend it for
testing. But Wget seems like a good project for testing. Do you have
any thoughts Bruno?

Collin

Re: gnulib-tool: Obey environment variable GNULIB_TOOL_IMPL

[Collin Funk]

Hi Simon,

On 3/15/24 7:30 AM, Simon Josefsson wrote:
> Wow it is very much faster!  \o/

That result is interesting. I wonder what part takes the shell script
so long. Maybe I'll use gnulib-tool as an excuse to expirement with
perf, dtrace, etc.

> OATH Toolkit fails with sh+py though...

Yes, Bruno made that check intentionally strict since we want the
output on the file system and stdout to be the same.

As a result though 'gnulib-tool --version' fails with
GNULIB_TOOL_IMPL=sh+py:

$ gnulib-tool.sh --version > shell.version
$ gnulib-tool.py --version > python.version
$ diff -u shell.version python.version 
--- shell.version   2024-03-15 11:07:43.096218254 -0700
+++ python.version  2024-03-15 11:07:50.895221724 -0700
@@ -4,4 +4,4 @@
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.
 
-Written by Bruno Haible, Paul Eggert, and Simon Josefsson.
+Written by Bruno Haible, Paul Eggert, Simon Josefsson, and Dmitry Selyutin.

which is slightly annoying. But better too strict than not strict
enough.

> isn't the --local-dir part working?  It doesn't notice the patches
> in liboath/gl/override.

I can build Coreutils and it uses --local-dir. It doesn't seem to use
patches though so that very well could be broken. Thanks for letting
me know. I'll look into it.

> Also it seems to "forget" gl_LGPL([2]) in gnulib-cache.m4.

I've run into a few "interesting" things with the output of the
gnulib-cache.m4 file. See:

$ gnulib-tool.py --create-testdir --dir test-python --vc-files dummy
$ cd test-python
$ gnulib-tool.py --import --vc-files readme-release
$ grep 'VC' m4/gnulib-cache.m4
gl_VC_FILES([True])

When fixing the conversion of shell 'true' to python 'True', it seems
to have uncovered that the conversion the other way around was never
implemented.

> From a clean checkout, you want to run this command to run all
> gnulib-tool invocations for OATH Toolkit:
> 
> make -f cfg.mk glimport

Thanks. This should help me out once I finish the gnulib-tool.py.TODO
items.

Collin

Re: gnulib-tool: Obey environment variable GNULIB_TOOL_IMPL

[Simon Josefsson via Gnulib discussion list]

Collin Funk  writes:

>> But in the current state, it fails for nearly every command. There's
>> no hope that you can expect identical results from the two implementations
>> as long as there are still items in the TODO list.
>
> Yes. I am working on it. I've added the following lines to my
> ~/.profile:
>
> GNULIB_TOOL_IMPL="sh+py"
> export GNULIB_TOOL_IMPL

Wow it is very much faster!  \o/

jas@kaka:~/src/oath-toolkit$ time env GNULIB_TOOL_IMPL=sh make -f cfg.mk 
glimport 
...
real0m44,908s
user0m47,285s
sys 0m8,306s
jas@kaka:~/src/oath-toolkit$ git clean -d -x -f; git restore --worktree 
--staged .
jas@kaka:~/src/oath-toolkit$ time env GNULIB_TOOL_IMPL=py make -f cfg.mk 
glimport 
...
real0m0,746s
user0m0,593s
sys 0m0,154s
jas@kaka:~/src/oath-toolkit$ 

OATH Toolkit fails with sh+py though...  isn't the --local-dir part
working?  It doesn't notice the patches in liboath/gl/override.  Also it
seems to "forget" gl_LGPL([2]) in gnulib-cache.m4.

From a clean checkout, you want to run this command to run all
gnulib-tool invocations for OATH Toolkit:

make -f cfg.mk glimport

/Simon


signature.asc
Description: PGP signature

[PATCH] gnulib-tool.py: Follow gnulib-tool changes, part 58.

[Collin Funk]

This patch implements --hardlink and --local-hardlink. It is a larger
patch so feel free to ask questions or let me know if I missed
anything.

Most of the changes are just following the referenced commit.

Outside of that, I noticed a bug when testing hard links. After
running gnulib-tool.py --import --hardlink module there was many files
with the name *.tmp suffix. I'm not sure if there was a patch that
changed the behavior or if it was a bug in the gnulib-tool.py
implementation. In any case, I solved this in
GLFileAssistant.add_or_update along with adding some comments taken
from gnulib-tool.sh.

I used the following script for testing:

gnulib-tool.py --hardlink --create-testdir --dir test-python readme-release
gnulib-tool.sh --hardlink --create-testdir --dir test-shell readme-release
(cd test-python && gnulib-tool.py -s --hardlink --symlink -h --import fts)
(cd test-shell && gnulib-tool.sh -s --hardlink --symlink -h --import fts)
git diff --no-index test-python test-shell

Notice the arguments '-s --hardlink --symlink -h'. I tried to make
sure gnulib-tool.py behaved similarly to gnulib-tool.sh in this case.

I beleive the correct behavior is to use the last option given so
'-h'. Which this seems to confirm:

$ stat lib/fts.c  | grep 'Links'
Device: 0,39Inode: 4852546 Links: 1
$ ./run-tests.sh
$ stat lib/fts.c  | grep 'Links'
Device: 0,39Inode: 4852546 Links: 3

It doesn't seem there is a good way to do this with Python's argparse,
though I may not have looked at the documentation hard enough [1]. I
ended up doing this which seems to work, though is not very pretty:

# --symlink and --hardlink are mutually exclusive.
# Use the last one given.
if symlink and hardlink:
optindex_table = {}
options = list(reversed(sys.argv[1:]))
options_count = len(options)
for option in ['-s', '--symbolic', '--symlink', '-h', '--hardlink']:
if option in options:
optindex_table[option] = options_count - options.index(option)
last_option = max(optindex_table, key=optindex_table.get)
# Disable the option that is not the last one.
if last_option in ['-s', '--symbolic', '--symlink']:
hardlink = False
else:  # last_option is --hardlink or equivalent.
symlink = False

And something similar for the --local-* variants.

I think that covers all of the strange hackery for this patch.

[1] https://docs.python.org/3/library/argparse.html

CollinFrom 3eeea5b376c5e0eceb5378bbbc3876ddbfbdca7c Mon Sep 17 00:00:00 2001
From: Collin Funk 
Date: Fri, 15 Mar 2024 06:38:48 -0700
Subject: [PATCH] gnulib-tool.py: Follow gnulib-tool changes, part 58.

Follow gnulib-tool change
2017-05-21  Bruno Haible  
gnulib-tool: Add options to create hard links.

* pygnulib/GLConfig.py (GLConfig.__init__): Add 'hardlink' and
'lhardlink' to the parameter list. Initialize them.
(GLConfig.default): Define the new options defaults to a boolean False.
Don't use links by default.
(GLConfig.checkHardlink, GLConfig.setHardlink, GLConfig.resetHardlink):
New functions to manipulate and check whether the --hardlink option was given.
(GLConfig.checkLHardlink, GLConfig.setLHardlink)
(GLConfig.resetLHardlink): New functions to manipulate and check whether
the --local-hardlink option was given.
* pygnulib/GLFileSystem.py (CopyAction.Hardlink): New Enum value to
describe hard links.
(GLFileSystem.shouldLink): Check if hard links should be used.
(GLFileAssistant.add, GLFileAssistant.update): Try to hard link if
enabled. Copy the file if linking fails.
(GLFileAssistant.add_or_update): Remove temporary files unconditionally.
* pygnulib/GLInfo.py (GLInfo.usage): Document new options in the usage
message.
* pygnulib/GLTestDir.py (GLTestDir.execute): Try to hard link if
enabled. Copy the file if linking fails.
* pygnulib/constants.py (hardlink): New function based on
symlink_relative.
* pygnulib/main.py (main): Add new options --hardlink and
--local-hardlink. Prefer the last given option when choosing symlinks or
hard links. Invoke 'git update-index --refresh' to mitigate the effects
of the hard links on git.
---
 ChangeLog| 32 ++
 gnulib-tool.py.TODO  | 31 +
 pygnulib/GLConfig.py | 53 +---
 pygnulib/GLFileSystem.py | 50 --
 pygnulib/GLInfo.py   |  4 +++
 pygnulib/GLTestDir.py|  4 ++-
 pygnulib/constants.py| 20 ++
 pygnulib/main.py | 59 
 8 files changed, 204 insertions(+), 49 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 08a8276629..f0d8f33956 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,35 @@
+2024-03-15  Collin Funk  
+
+	gnulib-tool.py: Follow gnulib-tool changes, part 58.
+	Follow gnulib-tool change
+	2017-05-21  Bruno Haible  
+	gnulib-tool: Add options to create hard links.
+	* pygnulib/GLConfig.py (GLConfig.__init__): 

Re: [PATCH] gnulib-tool.py: Follow gnulib-tool changes, part 57.

[Bruno Haible]

Collin Funk wrote:
> Oops, in the commit before I added some comments directly from
> gnulib-tool to main.py as well. Feel free to change them

No hurry, no problem. Such comments can stay as they are, in the short term.

Bruno