fast make install (was: Introduce and use the %KEY?TRUE:FALSE% template token)

[Ralf Wildenhues]

<http://lists.gnu.org/archive/html/automake-patches/2007-11/msg00057.html>

Hello Akim, and sorry for the delay,

* Akim Demaille wrote on Mon, Nov 19, 2007 at 06:47:41AM CET:
> 
> The first thing I would like to do is to factor the
> calls to install, using "install FILES... DIR".

Let's look at this in detail now:

0) Might I note that while history searching, I found that it was you
that opposed to the %KEY?TRUE:FALSE% template tokens earlier.  Hmm.  ;-)
<http://lists.gnu.org/archive/html/automake-patches/2001-10/msg00132.html>


1) The base case.  Current code looks like this:

install-includeHEADERS: $(include_HEADERS)
        @$(NORMAL_INSTALL)
        test -z "$(includedir)" || $(MKDIR_P) "$(DESTDIR)$(includedir)"
        @list='$(include_HEADERS)'; for p in $$list; do \
          if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \
          f=$(am__strip_dir) \
          echo " $(includeHEADERS_INSTALL) '$$d$$p' 
'$(DESTDIR)$(includedir)/$$f'"; \
           $(includeHEADERS_INSTALL) "$$d$$p" "$(DESTDIR)$(includedir)/$$f"; \
        done 

How would I rewrite it?

install-includeHEADERS: $(include_HEADERS)
        @$(NORMAL_INSTALL)
        test -z "$(includedir)" || $(MKDIR_P) "$(DESTDIR)$(includedir)"
        @list='$(include_HEADERS)'; list2=; for p in $$list; do \
          if test -f "$$p"; then d=; else d="$(srcdir)/"; fi; \
          list2="$$list2 '$$d$$p'"; \
        done; \
        echo " $(includeHEADERS_INSTALL) $$list2 '$(DESTDIR)$(includedir)'"; \
        $(includeHEADERS_INSTALL) $$list2 "$(DESTDIR)$(includedir)"

Looks pretty sane, except now building $list2 can scale quadratically in
the number of list items.  Oh well, most likely that will still be fast
enough (though we have found such issues to be limiting extreme cases in
libtool).  The systems that have high command line limits can typically
afford a better-behaved shell like bash 3.2+ as well.  For the others,
the above may newly run into the limits, if `$(srcdir)/' is long.

I suppose using a second dirname (includedir2 = $(includedir)) can be
employed by the user to work around remaining issues, if not then we
should revisit this.


2) Now to the nobase case:

> One problem is nobase_.  I see several means to cope with it:
> 
> - Don't try to factor calls to install with nobase_*

Ahh, no way.  ;-)

> - Add a special --nobase flag to install-sh and use it
>   blindly for nobase_*.

install-sh is blindly used for all nobase cases already,
in order to create the subdirectories needed (which install
doesn't create).  Which is the primary reason this is so slow.
Also, some BSD user complained they would like their install
binary to be used (which I think is a good idea).

Note an addition of --nobase is not enough, because you want
  install-sh --nobase a/b $srcdir/a/c $instdir

to strip srcdir.

> - Sort and filter the files to install in nobase_*.
>   I guess that in a first step, I would grep the files
>   that do not have a slash, and make one call for them,
>   then sort the remain files to group them per directory
>   and then make one call for each set.  This should be
>   simple using xargs, but I'm unsure about xargs'
>   portability.  Any clue?

Use awk to produce a sorted list.  Here's the code I came up
with, as a test case:

# test setup
srcdir=../S
DESTDIR=/tmp/dest
instdir=/usr/local/share
list='a b/c ../S/b/d b/e/f b/e/g b/e/g/h ../S/A A2 b/Ax b/X/A'

srcdirstrip=`echo "$srcdir" | sed 's/[].[^$\\*|]/\\\\&/g'`
for p in $list; do echo "$p $p"; done |
sed "s| $srcdirstrip/| |;"'
     / .*\//!s/ .*/ ./;
     s,\( .*\)/[^/]*$,\1,' |
awk '
BEGIN { files["."] = "" }
      { files[$2] = files[$2] " " $1
        if (++n[$2] == 50) {
          print $2, files[$2]
          n[$2] = 0
          files[$2] = ""
        }
      }
END   { for (dir in files)
          print dir " " files[dir]
      }' |
while read dir files; do
  xfiles=`for file in $files; do
            if test -f "$file"
              then echo "$file"
            else
              echo "$srcdir/$file"
            fi
          done`
  test -z "$xfiles" || {
    test "x$dir" = x. || { echo mkdir -p "$DESTDIR$instdir/$dir"; }
    echo install $xfiles "$DESTDIR$instdir/$dir"
  }
done


Needs 5 forks plus three per 50 files in a target directory, seems to
work on all kinds of systems.

One problem with this is that, if the user wants `install -m', it now
depends on awk hash order what permissions directories will have.
Should we sort the output of awk?

In my proposed patch (see my next mail), I have not converted
installation of SCRIPTS nor PROGRAMS because of $(transform).
I wonder whether it's useful to special-case the trivial case.

Aside, does anybody know why nobase.test requires gcc?

Cheers,
Ralf