Re: The non-recursive-gnulib-prefix-hack module

[Jim Meyering]

On Wed, Feb 1, 2017 at 2:28 PM, John W. Eaton  wrote:
> Is there anything I can do to help get this change accepted as part of
> gnulib?
>
> I'd really like to be able to use a completely non-recursive Makefile for
> Octave.

Hi John,
A great step in the right direction would be to ensure that your new
version works as well as the current one in projects like bison,
coreutils and grep that depend on it.

Re: The non-recursive-gnulib-prefix-hack module

[John W. Eaton]

Is there anything I can do to help get this change accepted as part of 
gnulib?


I'd really like to be able to use a completely non-recursive Makefile 
for Octave.


jwe


On 11/28/2016 12:40 AM, John W. Eaton wrote:

On 11/27/2016 06:18 PM, Bruno Haible wrote:

John,


I added

   AC_CONFIG_LIBOBJ_DIR([libgnu])

to my configure.ac file.  Maybe gnulib-tool could do this automatically
if --non-recursive-makefile is specified?


gnulib-tool does not rely on AC_LIBOBJ and friends, because these
autoconf
macros assume that there is only one lib/ dir, whereas it is possible to
have several gnulib-tool invocations in the scope of a single
configure.ac
file. Instead, the gnulib-tool generated gl_INIT method provides its own
definition of AC_LIBOBJ for the scope of the modules and .m4 expansions
that it pulls in.

Therefore you should not need this
AC_CONFIG_LIBOBJ_DIR([libgnu])
invocation.


I don't currently see how to make it work without using this macro.


it demonstrates what changes need to be made.


This is very good to see.


I'm attaching a new patch that fixes a few more things and seems to work
for me with Octave.


One thing I don't like is to put directory names into the *.m4 files;
this is
  1. against the principle that *.m4 files should determine properties
 of the platform, whereas Makefile.am contains file names,
  2. a problem w.r.t. the requirement above to be able to use
 several gnulib-tool invocations.

So, instead of replacing
LIMITS_H=limits.h
by
LIMITS_H=libgnu/limits.h
I would prefer to replace
AC_SUBST([LIMITS_H])
by
gl_FILENAME_SUBST([LIMITS_H])

Here gl_FILENAME_SUBST is a (yet to be written) autoconf macro that takes


If that's better, it should be easy to change, though I don't have any
more time to devote to this at the moment.

What I have now is

  LIMITS_H=${gl_REL_SOURCE_BASE}limits.h

With gl_REL_SOURCE_BASE defined to be "$sourcebase/" for non-recursive
builds or "" otherwise.


It should be done by gnulib-tool:
  1. because it can already be done at this stage,
  2. so that different gnulib-tool invocations in the scope of a single
 configure file work.


I think I have this working, at least for the Makefile.am snippets in
the module files now.


It seems like that would be easy enough just by arranging
to do something like

   gl_SOURCEBASE=$sourcebase
   AC_SUBST(gl_SOURCEBASE)

in the configure script and then using this variable in the snippets
where you used @FILE@ above.


There is a variable gl_source_base in the configure script, set at the
beginning of gl_INIT. You can use it as a shell variable. But it is not
(and cannot be) AC_SUBSTable.


I defined a separate variable because it needs to be either empty or
$sourcebase/ (with the /) and I believe that the existing variable
doesn't have the trailing slash.

Current patch attached.  Again, I only attempted to fix the module and
m4 files that are needed by Octave.

jwe

Re: The non-recursive-gnulib-prefix-hack module

[John W. Eaton]

On 11/27/2016 06:18 PM, Bruno Haible wrote:

John,


I added

   AC_CONFIG_LIBOBJ_DIR([libgnu])

to my configure.ac file.  Maybe gnulib-tool could do this automatically
if --non-recursive-makefile is specified?


gnulib-tool does not rely on AC_LIBOBJ and friends, because these autoconf
macros assume that there is only one lib/ dir, whereas it is possible to
have several gnulib-tool invocations in the scope of a single configure.ac
file. Instead, the gnulib-tool generated gl_INIT method provides its own
definition of AC_LIBOBJ for the scope of the modules and .m4 expansions
that it pulls in.

Therefore you should not need this
AC_CONFIG_LIBOBJ_DIR([libgnu])
invocation.


I don't currently see how to make it work without using this macro.


it demonstrates what changes need to be made.


This is very good to see.


I'm attaching a new patch that fixes a few more things and seems to work 
for me with Octave.



One thing I don't like is to put directory names into the *.m4 files;
this is
  1. against the principle that *.m4 files should determine properties
 of the platform, whereas Makefile.am contains file names,
  2. a problem w.r.t. the requirement above to be able to use
 several gnulib-tool invocations.

So, instead of replacing
LIMITS_H=limits.h
by
LIMITS_H=libgnu/limits.h
I would prefer to replace
AC_SUBST([LIMITS_H])
by
gl_FILENAME_SUBST([LIMITS_H])

Here gl_FILENAME_SUBST is a (yet to be written) autoconf macro that takes


If that's better, it should be easy to change, though I don't have any 
more time to devote to this at the moment.


What I have now is

  LIMITS_H=${gl_REL_SOURCE_BASE}limits.h

With gl_REL_SOURCE_BASE defined to be "$sourcebase/" for non-recursive 
builds or "" otherwise.



It should be done by gnulib-tool:
  1. because it can already be done at this stage,
  2. so that different gnulib-tool invocations in the scope of a single
 configure file work.


I think I have this working, at least for the Makefile.am snippets in 
the module files now.



It seems like that would be easy enough just by arranging
to do something like

   gl_SOURCEBASE=$sourcebase
   AC_SUBST(gl_SOURCEBASE)

in the configure script and then using this variable in the snippets
where you used @FILE@ above.


There is a variable gl_source_base in the configure script, set at the
beginning of gl_INIT. You can use it as a shell variable. But it is not
(and cannot be) AC_SUBSTable.


I defined a separate variable because it needs to be either empty or 
$sourcebase/ (with the /) and I believe that the existing variable 
doesn't have the trailing slash.


Current patch attached.  Again, I only attempted to fix the module and 
m4 files that are needed by Octave.


jwe

diff --git a/gnulib-tool b/gnulib-tool
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -253,6 +253,7 @@ Options for --import, --add/remove-impor
   --source-base=DIRECTORY
 Directory relative to --dir where source code is
 placed (default \"lib\").
+  --non-recursive-makefile  Generate non-recursive makefile snippet.
   --m4-base=DIRECTORY   Directory relative to --dir where *.m4 macros are
 placed (default \"m4\").
   --po-base=DIRECTORY   Directory relative to --dir where *.po files are
@@ -1011,6 +1012,7 @@ func_determine_path_separator
 # - verbose integer, default 0, inc/decremented by --verbose/--quiet
 # - libname, supplied_libname  from --lib
 # - sourcebase  from --source-base
+# - non_recursive   from --non-recursive-makefile
 # - m4base  from --m4-base
 # - pobase  from --po-base
 # - docbase from --doc-base
@@ -1062,6 +1064,8 @@ func_determine_path_separator
   libname=libgnu
   supplied_libname=
   sourcebase=
+  non_recursive=false
+  libamprefix=
   m4base=
   pobase=
   docbase=
@@ -1185,6 +1189,9 @@ func_determine_path_separator
 fi
 sourcebase=$1
 shift ;;
+  --non-recursive-makefile )
+non_recursive=true
+shift ;;
   --source-base=* )
 sourcebase=`echo "X$1" | sed -e 's/^X--source-base=//'`
 shift ;;
@@ -1427,6 +1434,10 @@ func_determine_path_separator
   echo "you need to use 'gnulib --import' - at your own risk!" 1>&2
   func_exit 1
 fi
+if $non_recursive && test -z "$makefile_name"; then
+  echo "you must use --makefile-name with --non-recursive-makefile" 1>2
+  func_exit 1
+fi
   fi
   if test -n "$pobase" && test -z "$po_domain"; then
 echo "gnulib-tool: together with --po-base, you need to specify 
--po-domain" 1>&2
@@ -2353,7 +2364,7 @@ func_get_automake_snippet_conditional ()
 {
   if ! $modcache; then
 func_lookup_file "modules/$1"
-sed -n -e "/^Makefile\.am$sed_extract_prog" < "$lookedup_file"
+sed -n -e "/^Makefile\.am$sed_extract_prog" < "$lookedup_file" | sed 
"s,@gl_REL_SOURCE_BASE@,${relsourcebase},g"
   else
 func_cache_lookup_module "$1"
 # 

Re: The non-recursive-gnulib-prefix-hack module

[Bruno Haible]

John,

> I added
> 
>AC_CONFIG_LIBOBJ_DIR([libgnu])
> 
> to my configure.ac file.  Maybe gnulib-tool could do this automatically 
> if --non-recursive-makefile is specified?

gnulib-tool does not rely on AC_LIBOBJ and friends, because these autoconf
macros assume that there is only one lib/ dir, whereas it is possible to
have several gnulib-tool invocations in the scope of a single configure.ac
file. Instead, the gnulib-tool generated gl_INIT method provides its own
definition of AC_LIBOBJ for the scope of the modules and .m4 expansions
that it pulls in.

Therefore you should not need this
AC_CONFIG_LIBOBJ_DIR([libgnu])
invocation.

> it demonstrates what changes need to be made.

This is very good to see.

One thing I don't like is to put directory names into the *.m4 files;
this is
  1. against the principle that *.m4 files should determine properties
 of the platform, whereas Makefile.am contains file names,
  2. a problem w.r.t. the requirement above to be able to use
 several gnulib-tool invocations.

So, instead of replacing
LIMITS_H=limits.h
by
LIMITS_H=libgnu/limits.h
I would prefer to replace
AC_SUBST([LIMITS_H])
by
gl_FILENAME_SUBST([LIMITS_H])

Here gl_FILENAME_SUBST is a (yet to be written) autoconf macro that takes
a variable as argument that contains a space-separated list of file names.
It prepends ${relsourcebase} to each of these file names. Then it uses
AC_SUBST to enable the substitution of @LIMITS_H@.

> It would also be great if there were some automake magic or if we 
> could fix gnulib-tool to do these things automatically, but I don't see 
> how at the moment.  I imagine that automatically modifying the contents 
> of things like EXTRA_DIST or BUILT_SOURCES might be possible, but I 
> don't see how to automatically adjust the hand-coded makefile snippets.

Right. When gnulib-tool generates only a part of Makefile.am, it must leave
some choices to the programmer. But we can print a message at the end of
the gnulib-tool run about what is recommended: the "Don't forget to"
section, around line 5750.

> > Yes, the lib_SOURCES augmentations (and likewise with EXTRA_DIST and 
> > probably
> > a few others as well) need to be edited, adding the value of 
> > ${relsourcebase}
> > in front of each file name.
> 
> I can look, but I'm not sure where to do that job.  If you can quickly 
> point to the spot where it needs to happen that will help, otherwise I 
> will do some searching.

Probably near func_get_automake_snippet.

> > Btw. does
> >SOURCEBASE/errno.h: SOURCEBASE/errno.in.h
> > work at all? Or do you need to write
> >SOURCEBASE/errno.h: $(srcdir)/SOURCEBASE/errno.in.h
> 
> I think it should using the VPATH rules and it does appear to work. 

Good. Fine.

> I assume if SOURCEBASE/errno.in.h was wrong, it would fail with some 
> message about not being able to find or make the prerequisite?

Yes.

> > This can safely be transformed through "s|@FILE@|SOURCEBASE|".
> 
> OK, I can try to make that change.  Are you thinking that this 
> transformation should be done by gnulib-tool, or using a configure 
> substitution?

It should be done by gnulib-tool:
  1. because it can already be done at this stage,
  2. so that different gnulib-tool invocations in the scope of a single
 configure file work.

> It seems like that would be easy enough just by arranging 
> to do something like
> 
>gl_SOURCEBASE=$sourcebase
>AC_SUBST(gl_SOURCEBASE)
> 
> in the configure script and then using this variable in the snippets 
> where you used @FILE@ above.

There is a variable gl_source_base in the configure script, set at the
beginning of gl_INIT. You can use it as a shell variable. But it is not
(and cannot be) AC_SUBSTable.

Bruno

Re: The non-recursive-gnulib-prefix-hack module

[John W. Eaton]

On 11/27/2016 05:02 PM, Bruno Haible wrote:


* I would add a check to make sure that --non-recursive-makefile is
  only used together with the --makefile-name option. I cannot see how/why
  someone would use --non-recursive-makefile without --makefile-name.
  This simplies some of the logic (e.g. the AUTOMAKE_OPTIONS handling).



* Code like this
if $non_recursive; then



* lt_prefix="${libname}_${libext}_"


OK, I can make these changes.


Yes, the lib_SOURCES augmentations (and likewise with EXTRA_DIST and probably
a few others as well) need to be edited, adding the value of ${relsourcebase}
in front of each file name.


I can look, but I'm not sure where to do that job.  If you can quickly 
point to the spot where it needs to happen that will help, otherwise I 
will do some searching.



Here the goal is to continue to keep the code in the module descriptions
as simple and straightforward as possible.


Yes, I agree.


This means that it needs to
be postprocessed. Such preprocessings already exist, e.g.
sed_replace_include_guard_prefix and sed_replace_build_aux are such
preprocessings, cf. function func_emit_autoconf_snippet.
Try to make the postprocessing as robust as possible. For example, a
substitution that replaces $(srcdir) by $(srcdir)/SOURCEBASE seems robust.

Btw. does
   SOURCEBASE/errno.h: SOURCEBASE/errno.in.h
work at all? Or do you need to write
   SOURCEBASE/errno.h: $(srcdir)/SOURCEBASE/errno.in.h


I think it should using the VPATH rules and it does appear to work. 
Here's what happens when I run make in a tree where nothing has changed 
in the sources:


  coredump:1358> make -j9
  make  all-am
  make[1]: Entering directory '/tmp/foo-build'

  Octave successfully built.  Now choose from the following:

 ./run-octave- to run in place to test before installing
 make check  - to run the tests
 make install- to install (PREFIX=/usr/local/octave/dev)

 HG ID for this build is "c88f88296789+"

  make[1]: Leaving directory '/tmp/foo-build'

I assume if SOURCEBASE/errno.in.h was wrong, it would fail with some 
message about not being able to find or make the prerequisite?


BTW, this is really awesome because now there is no recursive invocation 
of make to process anything, gnulib or otherwise, so it just finishes 
with the summary message and make finishes this job in about 9 seconds 
on my system instead of 18.



However, a substitution that replaces errno.h by SOURCEBASE/errno.h is
not robust. Probably we need some kind of annotation that can be
recognized by the preprocessing, e.g. let the snippet in the module description
read:

# We need the following in order to create  when the system
# doesn't have one that is POSIX compliant.
if GL_GENERATE_ERRNO_H
@FILE@errno.h: @f...@errno.in.h $(top_builddir)/config.status
$(AM_V_GEN)rm -f $@-t $@ && \
{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */' && \
  sed -e 's|@''GUARD_PREFIX''@|${gl_include_guard_prefix}|g' \
  [...] \
  < $(srcdir)/@f...@errno.in.h; \
} > $@-t && \
mv $@-t $@
else
@FILE@errno.h: $(top_builddir)/config.status
rm -f $@
endif

This can safely be transformed through "s|@FILE@|SOURCEBASE|".


OK, I can try to make that change.  Are you thinking that this 
transformation should be done by gnulib-tool, or using a configure 
substitution?  It seems like that would be easy enough just by arranging 
to do something like


  gl_SOURCEBASE=$sourcebase
  AC_SUBST(gl_SOURCEBASE)

in the configure script and then using this variable in the snippets 
where you used @FILE@ above.


jwe

Re: The non-recursive-gnulib-prefix-hack module

[John W. Eaton]

On 11/27/2016 04:29 PM, Bruno Haible wrote:


When you specify --makefile-name=module.mk, gnulib-tool knows that you
don't want a top-level Makefile.am and does not generate AUTOMAKE_OPTIONS
and generates augmentations (+=) of the other variables instead of assignments.


OK, I didn't realize that.  I also completely missed the option initially.

Now I'm calling gnulib-tool with the --makefile-name option and my new 
--non-recursive-makefile option.  Instead of using the 
non-recursive-gnulib-prefix-hack module I added


  AC_CONFIG_LIBOBJ_DIR([libgnu])

to my configure.ac file.  Maybe gnulib-tool could do this automatically 
if --non-recursive-makefile is specified?  I think it would be nice to 
just have the option (or a module) but not both.


I made some more modifications to gnulib-tool and the module and m4 
files that Octave uses.  Diffs are attached.


As you can see, I've hard-wired "libgnu" in many places.  Obviously this 
isn't the right thing to do, but it demonstrates what changes need to be 
made.  It would also be great if there were some automake magic or if we 
could fix gnulib-tool to do these things automatically, but I don't see 
how at the moment.  I imagine that automatically modifying the contents 
of things like EXTRA_DIST or BUILT_SOURCES might be possible, but I 
don't see how to automatically adjust the hand-coded makefile snippets.


jwe

diff --git a/gnulib-tool b/gnulib-tool
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -253,6 +253,7 @@ Options for --import, --add/remove-impor
   --source-base=DIRECTORY
 Directory relative to --dir where source code is
 placed (default \"lib\").
+  --non-recursive-makefile  Generate non-recursive makefile.
   --m4-base=DIRECTORY   Directory relative to --dir where *.m4 macros are
 placed (default \"m4\").
   --po-base=DIRECTORY   Directory relative to --dir where *.po files are
@@ -1011,6 +1012,7 @@ func_determine_path_separator
 # - verbose integer, default 0, inc/decremented by --verbose/--quiet
 # - libname, supplied_libname  from --lib
 # - sourcebase  from --source-base
+# - non_recursive   from --non-recursive-makefile
 # - m4base  from --m4-base
 # - pobase  from --po-base
 # - docbase from --doc-base
@@ -1062,6 +1064,8 @@ func_determine_path_separator
   libname=libgnu
   supplied_libname=
   sourcebase=
+  non_recursive=false
+  lt_prefix=
   m4base=
   pobase=
   docbase=
@@ -1185,6 +1189,9 @@ func_determine_path_separator
 fi
 sourcebase=$1
 shift ;;
+  --non-recursive-makefile )
+non_recursive=true
+shift ;;
   --source-base=* )
 sourcebase=`echo "X$1" | sed -e 's/^X--source-base=//'`
 shift ;;
@@ -2416,7 +2423,7 @@ func_get_automake_snippet_unconditional 
 | combine_lines \
 | sed -n -e "$sed_extract_mentioned_files" | sed -e 's/#.*//'`
   all_files=`func_get_filelist $1`
-  func_filter_filelist lib_files "$nl" "$all_files" 'lib/' '' 'lib/' ''
+  func_filter_filelist lib_files "$nl" "$all_files" 'lib/' '' 'lib/' '' 
"$sourcebase/" ''
   # Remove $already_mentioned_files from $lib_files.
   echo "$lib_files" | LC_ALL=C sort -u > "$tmp"/lib-files
   extra_files=`for f in $already_mentioned_files; do echo $f; done \
@@ -3423,6 +3430,17 @@ func_emit_lib_Makefile_am ()
 perhapsLT=
 sed_eliminate_LDFLAGS='/^lib_LDFLAGS[   ]*+=/d'
   fi
+  lt_prefix="${libname}_${libext}_"
+  if $non_recursive; then
+lt_prefix_tmp="`echo "$sourcebase" | LC_ALL=C sed -e 
's/[^a-zA-Z0-9_]/_/g'`"
+case "$lt_prefix_tmp" in
+  *_) ;;
+  *)
+lt_prefix_tmp="${lt_prefix_tmp}_"
+  ;;
+esac
+lt_prefix="$lt_prefix_tmp$lt_prefix"
+  fi
   if $for_test; then
 # When creating a package for testing: Attempt to provoke failures,
 # especially link errors, already during "make" rather than during
@@ -3457,19 +3475,19 @@ func_emit_lib_Makefile_am ()
 sed -e 's,lib_LIBRARIES,lib%_LIBRARIES,g' \
 -e 's,lib_LTLIBRARIES,lib%_LTLIBRARIES,g' \
 -e "$sed_eliminate_LDFLAGS" \
--e 's,lib_\([A-Z][A-Z]*\),'"${libname}_${libext}"'_\1,g' \
+-e 's,lib_\([A-Z][A-Z]*\),'"${lt_prefix}"'\1,g' \
 -e 's,lib%_LIBRARIES,lib_LIBRARIES,g' \
 -e 's,lib%_LTLIBRARIES,lib_LTLIBRARIES,g' \
 -e "$sed_transform_check_PROGRAMS" \
 -e "$sed_replace_include_guard_prefix"
   if test "$module" = 'alloca'; then
-echo "${libname}_${libext}_LIBADD += @${perhapsLT}ALLOCA@"
-echo "${libname}_${l

Re: The non-recursive-gnulib-prefix-hack module

[Bruno Haible]

Hi John,

> I also started working on a change to replace the 
> non-recursive-gnulib-prefix-hack module with a --non-recursive-makefile 
> option for gnulib-tool.  My initial attempt is attached.

Great!

A couple of small remarks:

* I would add a check to make sure that --non-recursive-makefile is
  only used together with the --makefile-name option. I cannot see how/why
  someone would use --non-recursive-makefile without --makefile-name.
  This simplies some of the logic (e.g. the AUTOMAKE_OPTIONS handling).

* Code like this
if $non_recursive; then
  echo "MOSTLYCLEANFILES $assign core *.stackdump"
else
  echo "MOSTLYCLEANFILES $assign ${sourcebase}core ${sourcebase}*.stackdump"
fi
  [btw. I think the then / else branches ought to be swapped]
  contains some code duplication. To avoid it, I would define a shell variable
  relsourcebase = (${sourcebase} if $non_recursive, "" otherwise).
  This will avoid code duplication.

* lt_prefix="${libname}_${libext}_"
  The prefix 'lt_' makes it sound as if it's something tied to libtool. Which
  is not the case here: this variable is useful also when building a static
  library (Automake doesn't even reference libtool in this case). I would rename
  it to libamprefix or automake_variables_prefix or something like that.

> I am stuck trying to figure out what to do about 
> the Makefile.am snippets that appear in module files.  I'm not sure 
> exactly where the file names in lines like
> 
>lib_SOURCES += xsize.c xsize.h
> 
> should be prefixed with the $sourcebase directory name or exactly how to 
> handle the rules that might be included there.

Yes, the lib_SOURCES augmentations (and likewise with EXTRA_DIST and probably
a few others as well) need to be edited, adding the value of ${relsourcebase}
in front of each file name.

> For example, I think things like
> 
># We need the following in order to create  when the system
># doesn't have one that is POSIX compliant.
>if GL_GENERATE_ERRNO_H
>errno.h: errno.in.h $(top_builddir)/config.status
>$(AM_V_GEN)rm -f $@-t $@ && \
>{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */' && \
>  sed -e 's|@''GUARD_PREFIX''@|${gl_include_guard_prefix}|g' \
>  [...] \
>  < $(srcdir)/errno.in.h; \
>} > $@-t && \
>mv $@-t $@
>else
>errno.h: $(top_builddir)/config.status
>rm -f $@
>endif
> 
> from modules/errno will need to have $(sourcebase) included in a few key 
> spots:
> 
># We need the following in order to create  when the system
># doesn't have one that is POSIX compliant.
>if GL_GENERATE_ERRNO_H
>SOURCEBASE/errno.h: SOURCEBASE/errno.in.h $(top_builddir)/config.status
>$(AM_V_GEN)rm -f $@-t $@ && \
>{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */' && \
>  sed -e 's|@''GUARD_PREFIX''@|${gl_include_guard_prefix}|g' \
>  [...] \
>  < $(srcdir)/SOURCEBASE/errno.in.h; \
>} > $@-t && \
>mv $@-t $@
>else
>SOURCEBASE/errno.h: $(top_builddir)/config.status
>rm -f $@
>endif

Here the goal is to continue to keep the code in the module descriptions
as simple and straightforward as possible. This means that it needs to
be postprocessed. Such preprocessings already exist, e.g.
sed_replace_include_guard_prefix and sed_replace_build_aux are such
preprocessings, cf. function func_emit_autoconf_snippet.
Try to make the postprocessing as robust as possible. For example, a
substitution that replaces $(srcdir) by $(srcdir)/SOURCEBASE seems robust.

Btw. does
   SOURCEBASE/errno.h: SOURCEBASE/errno.in.h
work at all? Or do you need to write
   SOURCEBASE/errno.h: $(srcdir)/SOURCEBASE/errno.in.h
?

However, a substitution that replaces errno.h by SOURCEBASE/errno.h is
not robust. Probably we need some kind of annotation that can be
recognized by the preprocessing, e.g. let the snippet in the module description
read:

# We need the following in order to create  when the system
# doesn't have one that is POSIX compliant.
if GL_GENERATE_ERRNO_H
@FILE@errno.h: @f...@errno.in.h $(top_builddir)/config.status
$(AM_V_GEN)rm -f $@-t $@ && \
{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */' && \
  sed -e 's|@''GUARD_PREFIX''@|${gl_include_guard_prefix}|g' \
  [...] \
  < $(srcdir)/@f...@errno.in.h; \
} > $@-t && \
mv $@-t $@
else
@FILE@errno.h: $(top_builddir)/config.status
rm -f $@
endif

This can safely be transformed through "s|@FILE@|SOURCEBASE|".

Bruno

Re: The non-recursive-gnulib-prefix-hack module

[Bruno Haible]

Hi John,

> I did that for Octave and the resulting files are attached.

This is great!

> The gnulib-generated file is Makefile.am.  The edited file that I 
> included in Octave's main Makefile.am file is module.mk.
> ...
> Things like AUTOMAKE_OPTIONS need to be omitted.  We can assume that the 
> top-level Makefile.am file will define that.
> 
> Variables like SUBDIRS, noinst_HEADERS, etc. don't need to be 
> initialized.  The top-level Makefile.am file should do that and the 
> makefile fragment that gnulib-tool generates should just append to these 
> variables.

When you specify --makefile-name=module.mk, gnulib-tool knows that you
don't want a top-level Makefile.am and does not generate AUTOMAKE_OPTIONS
and generates augmentations (+=) of the other variables instead of assignments.

> Additionally, I had to edit the definitions of gl_LIBOBJS and 
> gl_LTLIBOBJS in the generated Makefile to add the source-base prefix (we 
> use "libgnu/" in Octave).
> 
> The libgnu_la_FOO variables need to be libgnu_libgnu_la_FOO instead 
> (again assuming source-base is "libgnu/").
> 
> libgnu_libgnu_la_CPPFLAGS needs to be set so that libgnu/ headers are 
> correctly found in the source and build trees.
> 
> Source and target file names need to be adjusted so that they are found 
> in the libgnu/ directory.
> 
> The generated makefile fragment contains some things like @LTALLOCA@.  I 
> didn't do anything with those, but something will probably need to 
> change so that they have the proper directory prefix added to the 
> substituted values.
> 
> Rules like the ones that generate alloca.h, dirent.h, etc., need to be 
> fixed to find the source files in $(srcdir)/libgnu/FOO.in.h instead of 
> just $(srcdir)/FOO.in.h.

Yes, these are the things that gnulib-tool should do differently, when
the option --non-recursive-makefile is given.

> There were some instances of using a file name like t-$@.  That doesn't 
> work if $@ contains a directory component so I changed them to $@-t instead.

Yes, this is safe, even independently of the option --non-recursive-makefile.
(I had used t-$@ in order to avoid problems on file systems with limited
filename length, such as MINIX, FAT, or similar. But nowadays every file system
that people use for development has filename_max_length > 200.)

Bruno

Re: The non-recursive-gnulib-prefix-hack module

[John W. Eaton]

On 11/27/2016 07:52 AM, John W. Eaton wrote:

On 10/22/2016 03:04 PM, Bruno Haible wrote:


The idea would be to have gnulib-tool emit the correct code for the
{Bison,coreutils,Octave} case right away, triggered by some command line
option.

If you want to help us here, please use the *current* gnulib-tool to
generate Makefile.am files. Then hand-edit these Makefile.am files with
a minimum of changes, so that they work in a non-recursive build
(possibly
based on what prefix-gnulib-mk would produce). Then send us these files
(both the original and the edited Makefile.am) files, so that we can see
how gnulib-tool should be modified.


I did that for Octave and the resulting files are attached.


I also started working on a change to replace the 
non-recursive-gnulib-prefix-hack module with a --non-recursive-makefile 
option for gnulib-tool.  My initial attempt is attached.  It gets part 
of the way there, but I am stuck trying to figure out what to do about 
the Makefile.am snippets that appear in module files.  I'm not sure 
exactly where the file names in lines like


  lib_SOURCES += xsize.c xsize.h

should be prefixed with the $sourcebase directory name or exactly how to 
handle the rules that might be included there.  For example, I think 
things like


  # We need the following in order to create  when the system
  # doesn't have one that is POSIX compliant.
  if GL_GENERATE_ERRNO_H
  errno.h: errno.in.h $(top_builddir)/config.status
  $(AM_V_GEN)rm -f $@-t $@ && \
  { echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */' && \
sed -e 's|@''GUARD_PREFIX''@|${gl_include_guard_prefix}|g' \
[...] \
< $(srcdir)/errno.in.h; \
  } > $@-t && \
  mv $@-t $@
  else
  errno.h: $(top_builddir)/config.status
  rm -f $@
  endif

from modules/errno will need to have $(sourcebase) included in a few key 
spots:


  # We need the following in order to create  when the system
  # doesn't have one that is POSIX compliant.
  if GL_GENERATE_ERRNO_H
  SOURCEBASE/errno.h: SOURCEBASE/errno.in.h $(top_builddir)/config.status
  $(AM_V_GEN)rm -f $@-t $@ && \
  { echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */' && \
sed -e 's|@''GUARD_PREFIX''@|${gl_include_guard_prefix}|g' \
[...] \
< $(srcdir)/SOURCEBASE/errno.in.h; \
  } > $@-t && \
  mv $@-t $@
  else
  SOURCEBASE/errno.h: $(top_builddir)/config.status
  rm -f $@
  endif

I'm willing to continue working on this and/or testing with Octave but I 
will need a bit of guidance for how best to proceed.


Thanks,

jwe


diff --git a/gnulib-tool b/gnulib-tool
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -253,6 +253,7 @@ Options for --import, --add/remove-impor
   --source-base=DIRECTORY
 Directory relative to --dir where source code is
 placed (default \"lib\").
+  --non-recursive-makefile  Generate non-recursive makefile.
   --m4-base=DIRECTORY   Directory relative to --dir where *.m4 macros are
 placed (default \"m4\").
   --po-base=DIRECTORY   Directory relative to --dir where *.po files are
@@ -1011,6 +1012,7 @@ func_determine_path_separator
 # - verbose integer, default 0, inc/decremented by --verbose/--quiet
 # - libname, supplied_libname  from --lib
 # - sourcebase  from --source-base
+# - non_recursive   from --non-recursive-makefile
 # - m4base  from --m4-base
 # - pobase  from --po-base
 # - docbase from --doc-base
@@ -1062,6 +1064,8 @@ func_determine_path_separator
   libname=libgnu
   supplied_libname=
   sourcebase=
+  non_recursive=false
+  lt_prefix=
   m4base=
   pobase=
   docbase=
@@ -1185,6 +1189,9 @@ func_determine_path_separator
 fi
 sourcebase=$1
 shift ;;
+  --non-recursive-makefile )
+non_recursive=true
+shift ;;
   --source-base=* )
 sourcebase=`echo "X$1" | sed -e 's/^X--source-base=//'`
 shift ;;
@@ -2421,6 +2428,9 @@ func_get_automake_snippet_unconditional 
   echo "$lib_files" | LC_ALL=C sort -u > "$tmp"/lib-files
   extra_files=`for f in $already_mentioned_files; do echo $f; done \
| LC_ALL=C sort -u | LC_ALL=C join -v 2 - "$tmp"/lib-files`
+  if $non_recursive; then
+extra_files=`for f in $extra_files; do echo $sourcebase/$f; done`
+  fi
   if test -n "$extra_files"; then
 echo "EXTRA_DIST +=" $extra_files
 echo
@@ -3423,6 +3433,17 @@ func_emit_lib_Makefile_am ()
 perhapsLT=
 sed_eliminate_LDFLAGS='/^lib_LDFLAGS[   ]*+=/d'
   fi
+  lt_prefix="${libname}_${libext}_"
+  if $non_recursive; then
+lt_prefix_tmp="`echo "$sourcebase" | LC_ALL=C sed -e 
's/

Re: The non-recursive-gnulib-prefix-hack module

[Pádraig Brady]

On 07/11/16 02:35, Bruno Haible wrote:
> Hi Pádraig,
> 
>>> lib/gnulib.mk:32: error: required file '$gl_source_base/alloca.c' not found
>>> Makefile.am:226:   'lib/local.mk' included from here
>>> lib/local.mk:1:   'lib/gnulib.mk' included from here
>>> autoreconf: automake failed with exit status: 1
>>> ./bootstrap: autoreconf failed
> ...
>> The issue is in m4/non-recursive-gnulib-prefix-hack.m4
>> There are single quotes around $1 that need to be changed to double quotes.
> 
> This would not help, because at the point when
>   gl_LIBOBJS=`echo "$gl_LIBOBJS" | sed -e 's, , $gl_source_base/,g'`
> is being run, $gl_source_base has the value 'gnulib-tests', not 'lib'.
> 
>> More problematically AC_CONFIG_LIBOBJ_DIR is used there which
>> looks like it needs to be initialized before gl_source_base is set.
> 
> Yes, the statement
>   ac_config_libobj_dir=$gl_source_base
> is also out of order.
> 
>> I'll need to do a lot more digging to figure all that out.
> 
> It's probably not worth to try to fix the non-recursive-gnulib-prefix-hack
> module in this direction. Please feel free to revert my change to
> modules/non-recursive-gnulib-prefix-hack, and add a comment why 'lib' has
> to be hardcoded in this place.

Done.

thanks

Re: The non-recursive-gnulib-prefix-hack module

[Bruno Haible]

Hi Pádraig,

> > lib/gnulib.mk:32: error: required file '$gl_source_base/alloca.c' not found
> > Makefile.am:226:   'lib/local.mk' included from here
> > lib/local.mk:1:   'lib/gnulib.mk' included from here
> > autoreconf: automake failed with exit status: 1
> > ./bootstrap: autoreconf failed
...
> The issue is in m4/non-recursive-gnulib-prefix-hack.m4
> There are single quotes around $1 that need to be changed to double quotes.

This would not help, because at the point when
  gl_LIBOBJS=`echo "$gl_LIBOBJS" | sed -e 's, , $gl_source_base/,g'`
is being run, $gl_source_base has the value 'gnulib-tests', not 'lib'.

> More problematically AC_CONFIG_LIBOBJ_DIR is used there which
> looks like it needs to be initialized before gl_source_base is set.

Yes, the statement
  ac_config_libobj_dir=$gl_source_base
is also out of order.

> I'll need to do a lot more digging to figure all that out.

It's probably not worth to try to fix the non-recursive-gnulib-prefix-hack
module in this direction. Please feel free to revert my change to
modules/non-recursive-gnulib-prefix-hack, and add a comment why 'lib' has
to be hardcoded in this place.

Bruno

Re: The non-recursive-gnulib-prefix-hack module

[Pádraig Brady]

On 05/11/16 18:00, Pádraig Brady wrote:
> On 22/10/16 18:25, Bruno Haible wrote:
>> Hi Jim,
>>
>>>> 2016-10-16  Bruno Haible  <br...@clisp.org>
>>>>
>>>> gnulib-tool: Make --create-testdir on all modules work again.
>>>> * gnulib-tool (func_create_testdir): Don't include the
>>>> non-recursive-gnulib-prefix-hack module.
>>> That looks fine. Thanks again.
>>
>> OK, I've pushed it.
>>
>>>> 4) This module was added in September 2012. Does this mean that no one has 
>>>> used
>>>>gnulib-tool --create-testdir option to produce a testdir for all 
>>>> modules,
>>>>in four years?!
>>>
>>> Probably true.
>>
>> At least Tom Christensen is running gnulib-tool --create-testdir daily
>> (see http://lists.gnu.org/archive/html/bug-gnulib/2016-10/msg00085.html),
>> and he apparently started to exclude the module 
>> 'non-recursive-gnulib-prefix-hack'
>> when it started to introduce this problem.
>>
>>>> 1) Both modules/non-recursive-gnulib-prefix-hack and
>>>>m4/non-recursive-gnulib-prefix-hack.m4 hardcode the subdirectory name 
>>>> 'lib'.
>>>>(Why does m4/non-recursive-gnulib-prefix-hack.m4 use 'lib' in one place 
>>>> and $1
>>>>in another place??)
>>>
>>> That's a bug, but inconsequential. Thank you. I've just pushed a fix for it.
>>
>> This is better now. But still better would be to not hardcode 'lib' at all,
>> and instead use whatever value was passed to gnulib-tool via --source-base.
>> Below is a proposed patch. (The variable $gl_source_base is already used in a
>> similar way in the modules 'localcharset' and 'relocatable-prog'.)
>> Untested on my side; can you please test it the next time you happen to
>> build coreutils?
> 
> Unfortunately it fails with:
> 
> prefix=lib/
> running: AUTOPOINT=true LIBTOOLIZE=true autoreconf --verbose --install 
> --force -I m4  --no-recursive
> autoreconf: Entering directory `.'
> autoreconf: running: true --force
> autoreconf: running: aclocal -I m4 --force -I m4
> autoreconf: configure.ac: tracing
> autoreconf: configure.ac: not using Libtool
> autoreconf: running: /usr/bin/autoconf --include=m4 --force
> autoreconf: running: /usr/bin/autoheader --include=m4 --force
> autoreconf: running: automake --add-missing --copy --force-missing
> configure.ac:614: error: required directory ./$gl_source_base does not exist
> lib/gnulib.mk:32: error: required file '$gl_source_base/alloca.c' not found
> Makefile.am:226:   'lib/local.mk' included from here
> lib/local.mk:1:   'lib/gnulib.mk' included from here
> autoreconf: automake failed with exit status: 1
> ./bootstrap: autoreconf failed
> 
> 
>>
>> Bruno
>>
>>
>> 2016-10-22  Bruno Haible  <br...@clisp.org>
>>
>>  non-recursive-gnulib-prefix-hack: Don't make assumptions about
>>  gnulib-tool's --source-base option value.
>>  * modules/non-recursive-gnulib-prefix-hack (configure.ac): Use
>>  gl_source_base.
>>
>> diff --git a/modules/non-recursive-gnulib-prefix-hack 
>> b/modules/non-recursive-gnulib-prefix-hack
>> index 80bc27f..f938d30 100644
>> --- a/modules/non-recursive-gnulib-prefix-hack
>> +++ b/modules/non-recursive-gnulib-prefix-hack
>> @@ -12,7 +12,7 @@ configure.ac:
>>  dnl Run our hack near the end, just before config.status creation.
>>  dnl It must happen late, i.e., after gl_LIBOBJS has been finalized.
>>  AC_CONFIG_COMMANDS_PRE([
>> -  gl_NON_RECURSIVE_GNULIB_PREFIX_HACK([lib])
>> +  gl_NON_RECURSIVE_GNULIB_PREFIX_HACK([$gl_source_base])
>>])

The issue is in m4/non-recursive-gnulib-prefix-hack.m4
There are single quotes around $1 that need to be changed to double quotes.
More problematically AC_CONFIG_LIBOBJ_DIR is used there which
looks like it needs to be initialized before gl_source_base is set.

I'll need to do a lot more digging to figure all that out.

Re: The non-recursive-gnulib-prefix-hack module

[Pádraig Brady]

On 22/10/16 18:25, Bruno Haible wrote:
> Hi Jim,
> 
>>> 2016-10-16  Bruno Haible  <br...@clisp.org>
>>>
>>> gnulib-tool: Make --create-testdir on all modules work again.
>>> * gnulib-tool (func_create_testdir): Don't include the
>>> non-recursive-gnulib-prefix-hack module.
>> That looks fine. Thanks again.
> 
> OK, I've pushed it.
> 
>>> 4) This module was added in September 2012. Does this mean that no one has 
>>> used
>>>gnulib-tool --create-testdir option to produce a testdir for all modules,
>>>in four years?!
>>
>> Probably true.
> 
> At least Tom Christensen is running gnulib-tool --create-testdir daily
> (see http://lists.gnu.org/archive/html/bug-gnulib/2016-10/msg00085.html),
> and he apparently started to exclude the module 
> 'non-recursive-gnulib-prefix-hack'
> when it started to introduce this problem.
> 
>>> 1) Both modules/non-recursive-gnulib-prefix-hack and
>>>m4/non-recursive-gnulib-prefix-hack.m4 hardcode the subdirectory name 
>>> 'lib'.
>>>(Why does m4/non-recursive-gnulib-prefix-hack.m4 use 'lib' in one place 
>>> and $1
>>>in another place??)
>>
>> That's a bug, but inconsequential. Thank you. I've just pushed a fix for it.
> 
> This is better now. But still better would be to not hardcode 'lib' at all,
> and instead use whatever value was passed to gnulib-tool via --source-base.
> Below is a proposed patch. (The variable $gl_source_base is already used in a
> similar way in the modules 'localcharset' and 'relocatable-prog'.)
> Untested on my side; can you please test it the next time you happen to
> build coreutils?

Unfortunately it fails with:

prefix=lib/
running: AUTOPOINT=true LIBTOOLIZE=true autoreconf --verbose --install --force 
-I m4  --no-recursive
autoreconf: Entering directory `.'
autoreconf: running: true --force
autoreconf: running: aclocal -I m4 --force -I m4
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: running: /usr/bin/autoconf --include=m4 --force
autoreconf: running: /usr/bin/autoheader --include=m4 --force
autoreconf: running: automake --add-missing --copy --force-missing
configure.ac:614: error: required directory ./$gl_source_base does not exist
lib/gnulib.mk:32: error: required file '$gl_source_base/alloca.c' not found
Makefile.am:226:   'lib/local.mk' included from here
lib/local.mk:1:   'lib/gnulib.mk' included from here
autoreconf: automake failed with exit status: 1
./bootstrap: autoreconf failed


> 
> Bruno
> 
> 
> 2016-10-22  Bruno Haible  <br...@clisp.org>
> 
>   non-recursive-gnulib-prefix-hack: Don't make assumptions about
>   gnulib-tool's --source-base option value.
>   * modules/non-recursive-gnulib-prefix-hack (configure.ac): Use
>   gl_source_base.
> 
> diff --git a/modules/non-recursive-gnulib-prefix-hack 
> b/modules/non-recursive-gnulib-prefix-hack
> index 80bc27f..f938d30 100644
> --- a/modules/non-recursive-gnulib-prefix-hack
> +++ b/modules/non-recursive-gnulib-prefix-hack
> @@ -12,7 +12,7 @@ configure.ac:
>  dnl Run our hack near the end, just before config.status creation.
>  dnl It must happen late, i.e., after gl_LIBOBJS has been finalized.
>  AC_CONFIG_COMMANDS_PRE([
> -  gl_NON_RECURSIVE_GNULIB_PREFIX_HACK([lib])
> +  gl_NON_RECURSIVE_GNULIB_PREFIX_HACK([$gl_source_base])
>])
>  
>  Makefile.am:
> 
> 
> 

Re: The non-recursive-gnulib-prefix-hack module

[Bruno Haible]

Jim Meyering wrote:
> I've confirmed that with that, coreutils still bootstraps and passes
> "make distcheck".

Thank you for the testing, Jim. Pushed.

Bruno
--
In memoriam Hana Brady 

Re: The non-recursive-gnulib-prefix-hack module

[Jim Meyering]

On Sat, Oct 22, 2016 at 10:25 AM, Bruno Haible  wrote:
> This is better now. But still better would be to not hardcode 'lib' at all,
> and instead use whatever value was passed to gnulib-tool via --source-base.
> Below is a proposed patch. (The variable $gl_source_base is already used in a
> similar way in the modules 'localcharset' and 'relocatable-prog'.)
> Untested on my side; can you please test it the next time you happen to
> build coreutils?
>
> Bruno
>
>
> 2016-10-22  Bruno Haible  
>
> non-recursive-gnulib-prefix-hack: Don't make assumptions about
> gnulib-tool's --source-base option value.
> * modules/non-recursive-gnulib-prefix-hack (configure.ac): Use
> gl_source_base.
>
> diff --git a/modules/non-recursive-gnulib-prefix-hack 
> b/modules/non-recursive-gnulib-prefix-hack
> index 80bc27f..f938d30 100644
> --- a/modules/non-recursive-gnulib-prefix-hack
> +++ b/modules/non-recursive-gnulib-prefix-hack
> @@ -12,7 +12,7 @@ configure.ac:
>  dnl Run our hack near the end, just before config.status creation.
>  dnl It must happen late, i.e., after gl_LIBOBJS has been finalized.
>  AC_CONFIG_COMMANDS_PRE([
> -  gl_NON_RECURSIVE_GNULIB_PREFIX_HACK([lib])
> +  gl_NON_RECURSIVE_GNULIB_PREFIX_HACK([$gl_source_base])
>])
>
>  Makefile.am:

Nice improvement.
I've confirmed that with that, coreutils still bootstraps and passes
"make distcheck".
Thank you.

Re: The non-recursive-gnulib-prefix-hack module

[Mike Miller]

On Sat, Oct 22, 2016 at 19:25:46 +0200, Bruno Haible wrote:
> This is better now. But still better would be to not hardcode 'lib' at all,
> and instead use whatever value was passed to gnulib-tool via --source-base.
> Below is a proposed patch. (The variable $gl_source_base is already used in a
> similar way in the modules 'localcharset' and 'relocatable-prog'.)
> Untested on my side; can you please test it the next time you happen to
> build coreutils?

Hi, I had attempted to get non-recursive-gnulib-prefix-hack working with
Octave (where the gnulib subdirectory is not named 'lib'). I sent a
patch and a query for help to bug-gnulib last year and attracted no
interest, I assume because there are very few users.

Original (work-in-progress) patch attached, and see
http://lists.gnu.org/archive/html/bug-gnulib/2015-08/msg0.html,
still valid because Octave has still not managed to incorporate gnulib
cleanly into its non-recursive build.

I'd be grateful for any thoughts or help on making this module work more
generically.

-- 
mike
>From a017e33d890562e2bd456752345625c0bb8cc5f3 Mon Sep 17 00:00:00 2001
From: Mike Miller 
Date: Sat, 1 Aug 2015 12:25:48 -0400
Subject: [PATCH] non-recursive-gnulib-prefix-hack: fix when source-base !=
 "lib"

---
 build-aux/prefix-gnulib-mk | 3 ++-
 gnulib-tool| 4 
 m4/non-recursive-gnulib-prefix-hack.m4 | 2 +-
 3 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/build-aux/prefix-gnulib-mk b/build-aux/prefix-gnulib-mk
index 8af2916..3bb2a59 100755
--- a/build-aux/prefix-gnulib-mk
+++ b/build-aux/prefix-gnulib-mk
@@ -120,7 +120,8 @@ sub prefix_assignment ($$)
 
   # Variables which name depend on the location: libbison_a_SOURCES =>
   # lib_libbison_a_SOURCES.
-  $lhs_and_assign_op =~ s/($lib_name)/lib_$1/g;
+  my $dir = substr ($prefix, 0, -1);
+  $lhs_and_assign_op =~ s/($lib_name)/${dir}_$1/g;
 
   return $lhs_and_assign_op . $rhs;
 }
diff --git a/gnulib-tool b/gnulib-tool
index ec82f35..71e7d9f 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -3918,6 +3918,10 @@ func_emit_autoconf_snippet ()
 cat
   fi
 } \
+  | { if test "$sourcebase" != "lib"; then
+sed -e 's,\[lib\],'"[$sourcebase],g"
+  fi
+} \
   | { if $disable_gettext; then
 sed -e 's/AM_GNU_GETTEXT(\[external\])/dnl you must add AM_GNU_GETTEXT([external]) or similar to configure.ac./'
   else
diff --git a/m4/non-recursive-gnulib-prefix-hack.m4 b/m4/non-recursive-gnulib-prefix-hack.m4
index bf9ae7c..e827a52 100644
--- a/m4/non-recursive-gnulib-prefix-hack.m4
+++ b/m4/non-recursive-gnulib-prefix-hack.m4
@@ -10,7 +10,7 @@ dnl in the directory specified by LIB_DIR.
 AC_DEFUN([gl_NON_RECURSIVE_GNULIB_PREFIX_HACK],
 [
   # Tell AC_LIBSOURCES where to find source files like alloca.c.
-  AC_CONFIG_LIBOBJ_DIR([lib])
+  AC_CONFIG_LIBOBJ_DIR([$1])
 
   # This hack originated in bison.  It is required when using non-recursive
   # automake rules to build from gnulib-provided lib/ sources.  Hence, LIB_DIR
-- 
2.4.6

Re: The non-recursive-gnulib-prefix-hack module

[Bruno Haible]

Hi Mike,

> Hi, I had attempted to get non-recursive-gnulib-prefix-hack working with
> Octave (where the gnulib subdirectory is not named 'lib'). I sent a
> patch and a query for help to bug-gnulib last year and attracted no
> interest, I assume because there are very few users.
> 
> Original (work-in-progress) patch attached, and see
> http://lists.gnu.org/archive/html/bug-gnulib/2015-08/msg0.html,
> still valid because Octave has still not managed to incorporate gnulib
> cleanly into its non-recursive build.

The change to m4/non-recursive-gnulib-prefix-hack.m4 is no longer needed after
the two patches that I've submitted these days.

The change to gnulib-tool is not something I like: Replacing fixed strings
by variable strings a posteriori is a technique that produces good results
quickly but becomes more and more fragile the more it grows.

The change to build-aux/prefix-gnulib-mk is IMO lacking a conversion:
non-alphanumeric characters in $dir should be replaced by underscores. For
example, if $lib_name = "gnulib-lib", you don't want to produce a Makefile.am
variable name 'gnulib-lib_libbison_a_SOURCES' but rather
'gnulib_lib_libbison_a_SOURCES'.

> I'd be grateful for any thoughts or help on making this module work more
> generically.

The idea would be to have gnulib-tool emit the correct code for the
{Bison,coreutils,Octave} case right away, triggered by some command line
option.

If you want to help us here, please use the *current* gnulib-tool to
generate Makefile.am files. Then hand-edit these Makefile.am files with
a minimum of changes, so that they work in a non-recursive build (possibly
based on what prefix-gnulib-mk would produce). Then send us these files
(both the original and the edited Makefile.am) files, so that we can see
how gnulib-tool should be modified.

Bruno

Re: The non-recursive-gnulib-prefix-hack module

[Bruno Haible]

Hi Jim,

> > 2016-10-16  Bruno Haible  <br...@clisp.org>
> >
> > gnulib-tool: Make --create-testdir on all modules work again.
> > * gnulib-tool (func_create_testdir): Don't include the
> >     non-recursive-gnulib-prefix-hack module.
> That looks fine. Thanks again.

OK, I've pushed it.

> > 4) This module was added in September 2012. Does this mean that no one has 
> > used
> >gnulib-tool --create-testdir option to produce a testdir for all modules,
> >in four years?!
>
> Probably true.

At least Tom Christensen is running gnulib-tool --create-testdir daily
(see http://lists.gnu.org/archive/html/bug-gnulib/2016-10/msg00085.html),
and he apparently started to exclude the module 
'non-recursive-gnulib-prefix-hack'
when it started to introduce this problem.

> > 1) Both modules/non-recursive-gnulib-prefix-hack and
> >m4/non-recursive-gnulib-prefix-hack.m4 hardcode the subdirectory name 
> > 'lib'.
> >(Why does m4/non-recursive-gnulib-prefix-hack.m4 use 'lib' in one place 
> > and $1
> >in another place??)
> 
> That's a bug, but inconsequential. Thank you. I've just pushed a fix for it.

This is better now. But still better would be to not hardcode 'lib' at all,
and instead use whatever value was passed to gnulib-tool via --source-base.
Below is a proposed patch. (The variable $gl_source_base is already used in a
similar way in the modules 'localcharset' and 'relocatable-prog'.)
Untested on my side; can you please test it the next time you happen to
build coreutils?

Bruno


2016-10-22  Bruno Haible  <br...@clisp.org>

non-recursive-gnulib-prefix-hack: Don't make assumptions about
gnulib-tool's --source-base option value.
* modules/non-recursive-gnulib-prefix-hack (configure.ac): Use
gl_source_base.

diff --git a/modules/non-recursive-gnulib-prefix-hack 
b/modules/non-recursive-gnulib-prefix-hack
index 80bc27f..f938d30 100644
--- a/modules/non-recursive-gnulib-prefix-hack
+++ b/modules/non-recursive-gnulib-prefix-hack
@@ -12,7 +12,7 @@ configure.ac:
 dnl Run our hack near the end, just before config.status creation.
 dnl It must happen late, i.e., after gl_LIBOBJS has been finalized.
 AC_CONFIG_COMMANDS_PRE([
-  gl_NON_RECURSIVE_GNULIB_PREFIX_HACK([lib])
+  gl_NON_RECURSIVE_GNULIB_PREFIX_HACK([$gl_source_base])
   ])
 
 Makefile.am:

Re: The non-recursive-gnulib-prefix-hack module

[Jim Meyering]

On Sun, Oct 16, 2016 at 5:13 AM, Bruno Haible <br...@clisp.org> wrote:
> Hi Jim, all,
>
> When I try to use the gnulib-tool --create-testdir option to produce a testdir
> for all modules, it encounters errors and produces no toplevel Makefile.in:
>
> $ ./gnulib-tool --create-testdir --with-tests --dir=/tmp/testdir1
> gnulib-tool: warning: module euidaccess depends on a module with an 
> incompatible license: group-member
> gnulib-tool: warning: module strtod-obsolete depends on a module with an 
> incompatible license: strtod
> ...
> executing aclocal -I glm4
> executing autoconf
> executing autoheader
> executing automake --add-missing --copy
> configure.ac:3601: error: required directory ./lib does not exist
> configure.ac:23: installing 'build-aux/ar-lib'
> configure.ac:4: installing 'build-aux/install-sh'
> configure.ac:4: installing 'build-aux/missing'
> Makefile.am:5: warning: AM_GNU_GETTEXT used but 'po' not in SUBDIRS
> gllib/Makefile.am:47: error: required file 'lib/alloca.c' not found
> gllib/Makefile.am: installing 'build-aux/depcomp'
> configure.ac: installing 'build-aux/ylwrap'
>
> I've tracked it down to the module 'non-recursive-gnulib-prefix-hack':
>
> $ ./gnulib-tool --create-testdir --with-tests --dir=/tmp/testdir3 nocrash 
> non-recursive-gnulib-prefix-hack
> ...
> executing aclocal -I glm4
> executing autoconf
> executing autoheader
> executing automake --add-missing --copy
> configure.ac:109: error: required directory ./lib does not exist
> ...
>
> $ ./gnulib-tool --create-testdir --with-tests --dir=/tmp/testdir3 alloca 
> non-recursive-gnulib-prefix-hack
> ...
> executing aclocal -I glm4
> executing autoconf
> executing autoheader
> executing automake --add-missing --copy
> configure.ac:110: error: required directory ./lib does not exist
> configure.ac:22: installing 'build-aux/ar-lib'
> configure.ac:8: installing 'build-aux/compile'
> configure.ac:4: installing 'build-aux/install-sh'
> configure.ac:4: installing 'build-aux/missing'
> automake: error: '$(ALLOCA)' cannot be used outside 'lib' if 'subdir-objects' 
> is not set
> gllib/Makefile.am:46: error: required file 'lib/alloca.c' not found
> gllib/Makefile.am: installing 'build-aux/depcomp'
>
> Here are the problems that I see:
>
> 1) Both modules/non-recursive-gnulib-prefix-hack and
>m4/non-recursive-gnulib-prefix-hack.m4 hardcode the subdirectory name 
> 'lib'.
>(Why does m4/non-recursive-gnulib-prefix-hack.m4 use 'lib' in one place 
> and $1
>in another place??)

That's a bug, but inconsequential. Thank you. I've just pushed a fix for it.

> 2) This is a different way of using Automake; it should not be used in 
> gnulib-tool's
>testdirs.

Thanks for working around it.

> 3) Why has this feature been implemented as a module that hacks around in
>undocumented Autoconf internals (ac_subst_vars is undocumented, no?, and
>gl_LIBOBJS is undocumented as well), rather than as a command-line option
>to gnulib-tool?

It is all a big hack, as the name suggests.
Bison found a way to use automake that way, and since it was nontrivial
to get right and I wanted to do the same for coreutils, I encapsulated the
ugliness in this macro.

> 4) This module was added in September 2012. Does this mean that no one has 
> used
>gnulib-tool --create-testdir option to produce a testdir for all modules,
>in four years?!

Probably true.
Recently I tried to do something similar, but got side-tracked.
I'm glad you're fixing all of this.

> For problem 2), I propose the remedy below.
>
> Bruno
>
>
> 2016-10-16  Bruno Haible  <br...@clisp.org>
>
> gnulib-tool: Make --create-testdir on all modules work again.
> * gnulib-tool (func_create_testdir): Don't include the
> non-recursive-gnulib-prefix-hack module.
>
> diff --git a/gnulib-tool b/gnulib-tool
> index 12ee19f..666057a 100755
> --- a/gnulib-tool
> +++ b/gnulib-tool
> @@ -5811,10 +5811,12 @@ func_create_testdir ()
>if test -z "$modules"; then
>  # All modules together.
>  # Except config-h, which breaks all modules which use HAVE_CONFIG_H.
> +# Except non-recursive-gnulib-prefix-hack, which represents a nonstandard
> +# way of using Automake.
>  # Except ftruncate, mountlist, which abort the configuration on mingw. 
> FIXME.
>  # Except lib-ignore, which leads to link errors when Sun C++ is used. 
> FIXME.
>  modules=`func_all_modules`
> -modules=`for m in $modules; do case $m in config-h | ftruncate | 
> mountlist | lib-ignore) ;; *) echo $m;; esac; done`
> +modules=`for m in $modules; do case $m in config-h | 
> non-recursive-gnulib-prefix-hack | ftruncate | mountlist | lib-ignore) ;; *) 
> echo $m;; esac; done`
>fi
>specified_modules="$modules"

That looks fine. Thanks again.

The non-recursive-gnulib-prefix-hack module

[Bruno Haible]

Hi Jim, all,

When I try to use the gnulib-tool --create-testdir option to produce a testdir
for all modules, it encounters errors and produces no toplevel Makefile.in:

$ ./gnulib-tool --create-testdir --with-tests --dir=/tmp/testdir1
gnulib-tool: warning: module euidaccess depends on a module with an 
incompatible license: group-member
gnulib-tool: warning: module strtod-obsolete depends on a module with an 
incompatible license: strtod
...
executing aclocal -I glm4
executing autoconf
executing autoheader
executing automake --add-missing --copy
configure.ac:3601: error: required directory ./lib does not exist
configure.ac:23: installing 'build-aux/ar-lib'
configure.ac:4: installing 'build-aux/install-sh'
configure.ac:4: installing 'build-aux/missing'
Makefile.am:5: warning: AM_GNU_GETTEXT used but 'po' not in SUBDIRS
gllib/Makefile.am:47: error: required file 'lib/alloca.c' not found
gllib/Makefile.am: installing 'build-aux/depcomp'
configure.ac: installing 'build-aux/ylwrap'

I've tracked it down to the module 'non-recursive-gnulib-prefix-hack':

$ ./gnulib-tool --create-testdir --with-tests --dir=/tmp/testdir3 nocrash 
non-recursive-gnulib-prefix-hack
...
executing aclocal -I glm4
executing autoconf
executing autoheader
executing automake --add-missing --copy
configure.ac:109: error: required directory ./lib does not exist
...

$ ./gnulib-tool --create-testdir --with-tests --dir=/tmp/testdir3 alloca 
non-recursive-gnulib-prefix-hack
...
executing aclocal -I glm4
executing autoconf
executing autoheader
executing automake --add-missing --copy
configure.ac:110: error: required directory ./lib does not exist
configure.ac:22: installing 'build-aux/ar-lib'
configure.ac:8: installing 'build-aux/compile'
configure.ac:4: installing 'build-aux/install-sh'
configure.ac:4: installing 'build-aux/missing'
automake: error: '$(ALLOCA)' cannot be used outside 'lib' if 'subdir-objects' 
is not set
gllib/Makefile.am:46: error: required file 'lib/alloca.c' not found
gllib/Makefile.am: installing 'build-aux/depcomp'

Here are the problems that I see:

1) Both modules/non-recursive-gnulib-prefix-hack and
   m4/non-recursive-gnulib-prefix-hack.m4 hardcode the subdirectory name 'lib'.
   (Why does m4/non-recursive-gnulib-prefix-hack.m4 use 'lib' in one place and 
$1
   in another place??)

2) This is a different way of using Automake; it should not be used in 
gnulib-tool's
   testdirs.

3) Why has this feature been implemented as a module that hacks around in
   undocumented Autoconf internals (ac_subst_vars is undocumented, no?, and
   gl_LIBOBJS is undocumented as well), rather than as a command-line option
   to gnulib-tool?

4) This module was added in September 2012. Does this mean that no one has used
   gnulib-tool --create-testdir option to produce a testdir for all modules,
   in four years?!

For problem 2), I propose the remedy below.

Bruno


2016-10-16  Bruno Haible  <br...@clisp.org>

gnulib-tool: Make --create-testdir on all modules work again.
* gnulib-tool (func_create_testdir): Don't include the
        non-recursive-gnulib-prefix-hack module.

diff --git a/gnulib-tool b/gnulib-tool
index 12ee19f..666057a 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -5811,10 +5811,12 @@ func_create_testdir ()
   if test -z "$modules"; then
 # All modules together.
 # Except config-h, which breaks all modules which use HAVE_CONFIG_H.
+# Except non-recursive-gnulib-prefix-hack, which represents a nonstandard
+# way of using Automake.
 # Except ftruncate, mountlist, which abort the configuration on mingw. 
FIXME.
 # Except lib-ignore, which leads to link errors when Sun C++ is used. 
FIXME.
 modules=`func_all_modules`
-modules=`for m in $modules; do case $m in config-h | ftruncate | mountlist 
| lib-ignore) ;; *) echo $m;; esac; done`
+modules=`for m in $modules; do case $m in config-h | 
non-recursive-gnulib-prefix-hack | ftruncate | mountlist | lib-ignore) ;; *) 
echo $m;; esac; done`
   fi
   specified_modules="$modules"