Re: pygnulib: progress and some questions (func_emit_autoconf_snippets)

[Dmitry Selyutin]

Hi Bruno,

thank you for the detailed answer. I think I still have to do some
debugging to understand why the Python version of the algorithm for
autoconf snippets generation works incorrectly, but I hope I understood how
it should work.

29 дек. 2017 г. 2:32 ДП пользователь "Bruno Haible" 
написал:

> Hi Dmitry,
>
> > > 1. At line 4260, what's the place where $modules variable gets
> assigned?
> >
> > Indeed the comments for function func_emit_autoconf_snippets are
> misleading:
> > This functions receives two module lists:
> >   $modules (implicit),
> >   $1 (explicit: first argument).
> >
> > Where does $modules come from?
> >   - For the invocation in func_import, line 5640 and 5659
> > it is set at line 5298. But I'm not sure this was intended this way,
> maybe
> > another list should be used instead of "$main_modules"?
> >   - For the invocation in func_create_testdir, line 6288 and 6290 and
> 6402 and 6404 and 6417
> > it does not appear to be set in a consistent and correct way.
> > (It is set differently if $single_configure and ! $single_configure.)
> > This is probably a bug.
> > The set of modules that _ought_ to be used here is the set of modules
> > being used for the current configure file. That's my current
> understanding.
> > But I'd really need to look at a couple of test cases in order to
> > firmly answer the question.
>
> I've understood it now: the code in func_create_testdir was not really a
> bug,
> because --conditional-dependencies was not allowed in this case anyway.
>
> Here's the patch that implements --conditional-dependencies also with
> --with-tests.
>
> Bruno
>

Re: pygnulib: progress and some questions (func_emit_autoconf_snippets)

[Bruno Haible]

Hi Dmitry,

> > 1. At line 4260, what's the place where $modules variable gets assigned?
> 
> Indeed the comments for function func_emit_autoconf_snippets are misleading:
> This functions receives two module lists:
>   $modules (implicit),
>   $1 (explicit: first argument).
> 
> Where does $modules come from?
>   - For the invocation in func_import, line 5640 and 5659
> it is set at line 5298. But I'm not sure this was intended this way, maybe
> another list should be used instead of "$main_modules"?
>   - For the invocation in func_create_testdir, line 6288 and 6290 and 6402 
> and 6404 and 6417
> it does not appear to be set in a consistent and correct way.
> (It is set differently if $single_configure and ! $single_configure.)
> This is probably a bug.
> The set of modules that _ought_ to be used here is the set of modules
> being used for the current configure file. That's my current understanding.
> But I'd really need to look at a couple of test cases in order to
> firmly answer the question.

I've understood it now: the code in func_create_testdir was not really a bug,
because --conditional-dependencies was not allowed in this case anyway.

Here's the patch that implements --conditional-dependencies also with
--with-tests.

Bruno
>From 589e96475f8f2d21a83405ab0672ce95091b80e5 Mon Sep 17 00:00:00 2001
From: Bruno Haible 
Date: Fri, 29 Dec 2017 00:29:23 +0100
Subject: [PATCH] gnulib-tool: Make --conditional-dependencies work better.

Reported by Dmitry Selyutin .

* gnulib-tool (Options): Don't reject the combination of
--conditional-dependencies with --with-tests.
(func_emit_autoconf_snippets): Add argument referenceable_modules.
Don't reference $modules.
(func_import, func_create_testdir): Pass it.
---
 ChangeLog   | 10 ++
 gnulib-tool | 37 -
 2 files changed, 30 insertions(+), 17 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 7d7b10a..1e2e0fb 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,13 @@
+2017-12-28  Bruno Haible  
+
+	gnulib-tool: Make --conditional-dependencies work better.
+	Reported by Dmitry Selyutin .
+	* gnulib-tool (Options): Don't reject the combination of
+	--conditional-dependencies with --with-tests.
+	(func_emit_autoconf_snippets): Add argument referenceable_modules.
+	Don't reference $modules.
+	(func_import, func_create_testdir): Pass it.
+
 2017-12-19  Paul Eggert  
 
 	regex: use re_malloc etc. consistently
diff --git a/gnulib-tool b/gnulib-tool
index 22817c9..5f640e3 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -1505,10 +1505,6 @@ func_determine_path_separator
   esac
   # Now the only possible values of "$inctests" are true and false
   # (or blank but then it is irrelevant).
-  if test "$cond_dependencies" = true && test "$inctests" = true; then
-echo "gnulib-tool: option --conditional-dependencies is not supported with --with-tests" 1>&2
-func_exit 1
-  fi
 
   # Determine the minimum supported autoconf version from the project's
   # configure.ac.
@@ -4231,7 +4227,7 @@ func_emit_autoconf_snippet ()
   fi
 }
 
-# func_emit_autoconf_snippets modules verifier toplevel disable_libtool disable_gettext
+# func_emit_autoconf_snippets modules referenceable_modules verifier toplevel disable_libtool disable_gettext
 # collects and emit the autoconf snippets of a set of modules.
 # Input:
 # - local_gnulib_path  from --local-dir
@@ -4240,6 +4236,7 @@ func_emit_autoconf_snippet ()
 # - sed_replace_include_guard_prefix
 # sed expression for resolving ${gl_include_guard_prefix}
 # - modules   the list of modules.
+# - referenceable_modules  the list of modules which may be referenced as dependencies.
 # - verifier  one of func_verify_module, func_verify_nontests_module,
 # func_verify_tests_module. It selects the subset of
 # $modules to consider.
@@ -4252,12 +4249,13 @@ func_emit_autoconf_snippet ()
 # invocations.
 func_emit_autoconf_snippets ()
 {
-  verifier="$2"
-  toplevel="$3"
-  disable_libtool="$4"
-  disable_gettext="$5"
+  referenceable_modules="$2"
+  verifier="$3"
+  toplevel="$4"
+  disable_libtool="$5"
+  disable_gettext="$6"
   if test "$cond_dependencies" = true; then
-for m in $modules; do echo $m; done | LC_ALL=C sort -u > "$tmp"/modules
+for m in $referenceable_modules; do echo $m; done | LC_ALL=C sort -u > "$tmp"/modules
 # Emit the autoconf code for the unconditional modules.
 for module in $1; do
   eval $verifier
@@ -5637,7 +5635,7 @@ s,//*$,/,'
 if test -n "$witness_c_macro"; then
   echo "  m4_pushdef([gl_MODULE_INDICATOR_CONDITION], [$witness_c_macro])"
 fi
-func_emit_autoconf_snippets "$main_modules" func_verify_module true false true
+func_emit_autoconf_snippets "$main_modules" "$main_modules" func_verify_module true false true
 if test 

Re: pygnulib: progress and some questions (func_emit_autoconf_snippets)

[Bruno Haible]

Hi Dmitry,

> 1. At line 4260, what's the place where $modules variable gets assigned?

Indeed the comments for function func_emit_autoconf_snippets are misleading:
This functions receives two module lists:
  $modules (implicit),
  $1 (explicit: first argument).

Where does $modules come from?
  - For the invocation in func_import, line 5640 and 5659
it is set at line 5298. But I'm not sure this was intended this way, maybe
another list should be used instead of "$main_modules"?
  - For the invocation in func_create_testdir, line 6288 and 6290 and 6402 and 
6404 and 6417
it does not appear to be set in a consistent and correct way.
(It is set differently if $single_configure and ! $single_configure.)
This is probably a bug.
The set of modules that _ought_ to be used here is the set of modules
being used for the current configure file. That's my current understanding.
But I'd really need to look at a couple of test cases in order to
firmly answer the question.

> 2. What's the logic for dependencies processing, starting at line 4298?

First the code intersects $deps with $modules. This matters for two cases:
  - when --avoid is used: we don't want to invoke the initialization code for
a module that is not present because the user has required it to be avoided.
  - when ! $single_configure and we are generating the configure file for the 
tests
directory: we may have a dependency from a module in $tests_modules to a 
module
in $main_modules, but since the $main_modules are in a different directory
they should be all treated like --avoid'ed modules here.

Then the test
if func_cond_module_p "$dep"; then
is based on the consideration that the unconditional modules have their
initialization code already emitted (unconditionally) by lines 4261..4271.

Then comes the conditional invocation of the initialization code
echo "  if $condition; then"
echo "$shellfunc"
echo '  fi'
with a trivial optimization for the (frequent) case that $condition = "true".

> 3. Do you have an idea how could the algorithm mapped to the described API

You need db.dependencies($module) here. db.demanders(...) is not used here.
db.demanders(...) will be useful, in order to implement a new command-line
option --extract-reverse-dependencies.

Bruno

pygnulib: progress and some questions (func_emit_autoconf_snippets)

[Dmitry Selyutin]

Hello,

this is a kind reminder that python branch is still alive, though it's quite 
difficult to find a time 
for coding. :-) Anyway, I think I'll finish the "import" section in about a 
month: right now I'm 
slowly processing gnulib-comp.m4 generation part of the code. I have a question 
regarding a 
function named func_emit_autoconf_snippets.

TL;DR: feel free to skip to ===QUESTIONS=== section if this is a way too long 
introduction to 
the topic. :-) The questions mainly target Bruno as the original author, but 
I'd be very grateful 
for any help.


Could you please explain in details how func_emit_autoconf_snippets works? The 
most 
important for me is to understand how to map it into the API I've recently 
implemented: 
currently pygnulib can create a special object, "module database", which allows 
to iterate over 
modules and dependencies very quickly. Basically it is a hash table where each 
entry contains 
yet another hash table.


Given a dependency DEP, it is really easy to iterate over each demander DEM and 
condition 
COND like this:

for (dem, cond) in db.demanders(dep):
print(dem, cond)

One can also iterate over all dependencies of the demander DEM, acquiring each 
dependency 
DEP and the condition COND:

for (dep, cond) in db.dependencies(dem):
print(dep, cond)

So given a set of modules MODULES, I can hopefully obtain all the required 
information to 
implement func_emit_autoconf_snippets analogue. However, I can't quite 
understand how the 
modules are processed. The questions are followed; they all are relevant only 
for --
conditional-dependencies switch since otherwise the algorithm is trivial (just 
treat each 
module as unconditional).


===QUESTIONS===
1. At line 4260, what's the place where $modules variable gets assigned? The 
nearest explicit 
assignment I've found is at line 5298; is it correct that this is the one whose 
result is used at 
4260?
2. What's the logic for dependencies processing, starting at line 4298?
3. Do you have an idea how could the algorithm mapped to the described API, 
given that I 
have module database, main modules, tests modules and final modules (and 
hopefully every 
other bit of information)? There's no need to check whether module exists, 
since all the 
mentioned collections (database and modules lists) contain not strings, but a 
real module 
instances.


Thank you for your help!


-- 
With best regards,
Dmitry Selyutin



signature.asc
Description: This is a digitally signed message part.