bug#31157: Advice for help2man does not work for parallel builds

[Mathieu Lirzin]

Hello Peter,

Peter Johansson  writes:

> On 4/22/2018 1:13 AM, Mathieu Lirzin wrote:
>> Hello Reuben,
>>
>> Reuben Thomas  writes:
>>
>>> In the manual, we are given the following pattern for using help2man
>>> without breaking make distcheck:
>>>
>>> foo.1: foo.c $(top_srcdir)/configure.ac
>>> $(MAKE) $(AM_MAKEFLAGS) foo$(EXEEXT)
>>> help2man --output=foo.1 ./foo$(EXEEXT)
>>>
>>> The problem is that with make -j this can result in two attempts to
>>> make a library in parallel (suppose that we have:
>>>
>>> foo_LDADD = libfoo.la
>>> lib_LTLIBRARIES = libfoo.la
>>>
>>> ). This can fail, and in any case is wasteful.
>> Have you identified the reason why this can fail?  because
>
> One problem is that the rule for foo.1 can be triggered before
> foo.$(EXEEXT) exists and the rule needs foo.$(EXEEXT) or help2man will
> fail. In a -j1 build this is never a problem as binaries are built
> before man pages.

with the ‘$(MAKE) $(AM_MAKEFLAGS) foo$(EXEEXT)’ recipe line, normally
this ensures that the ‘foo$(EXEEXT)’ is built at least once.  My
question was about understanding why building a program/library twice
may end up in a failure in a parallel context.

The answer from Reuben was that since compilation is not thead safe when
for example deleting shared temporary files, then concurrent processes
might impact each other.

> When the 'missing' script changed behaviour in Automake 1.13 (and
> became useless imvho), we changed the rule in one project so foo.1
> depended on foo.$(EXEEXT) but not if we are 1) building from a tarball
> and 2) foo.$(EXEEXT) exists.
>
> http://dev.thep.lu.se/svndigest/browser/trunk/man/Makefile.am

This is an alternative solution which is used by GNU Hello too.  I
remember some people not being fond of this approach because of its
AM_MAINTAINER_MODE spirit [1].

[1] 
https://www.gnu.org/software/automake/manual/html_node/maintainer_002dmode.html

Thanks for your feedback.

-- 
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

bug#31157: Advice for help2man does not work for parallel builds

[Mathieu Lirzin]

Reuben Thomas  writes:

> On 21 April 2018 at 16:13, Mathieu Lirzin  wrote:
>
>  Reuben Thomas  writes:
>
>  > In the manual, we are given the following pattern for using help2man
>  > without breaking make distcheck:
>  >
>  > foo.1: foo.c $(top_srcdir)/configure.ac
>  > $(MAKE) $(AM_MAKEFLAGS) foo$(EXEEXT)
>  > help2man --output=foo.1 ./foo$(EXEEXT)
>  >
>  > The problem is that with make -j this can result in two attempts to
>  > make a library in parallel (suppose that we have:
>  >
>  > foo_LDADD = libfoo.la
>  > lib_LTLIBRARIES = libfoo.la
>  >
>  > ). This can fail, and in any case is wasteful.
>
>  Have you identified the reason why this can fail? because 
>
> ​Because two independent parallel invocations of make ​can end up trying
> to build the library (which is wasteful anyway) and some needed file
> can be deleted by one invocation when the other is trying to use it to
> link the library.

Makes sense.

>  This is not ideal since this result in making ‘help2man’ (and ‘perl’
>  transitively) a build dependency for tarball builders.
>
> ​I'm increasingly of the view this is not a problem. Perl is
> increasingly reasonable as a build dep (it seems to be in most base
> systems now), and help2man is small.

In term of convenience for end users, I agree this is often a harmless
issue.  However having been involved in GNU Guix for awhile I think it
is important to avoid accidental dependencies to improve the simplicity
of the bootstrapping process of our systems [1][2].

>  $(srcdir)/foo.1: foo.c foo$(EXEEXT)
>  -@case '$?' in \
>  *foo.c*) 
>  ​​$(AM_V_P) && set -x || echo " HELP2MAN $@"; \
>  LANGUAGE= help2man --output="$(srcdir)/foo.1" ./foo$(EXEEXT);; \
>  *) : ;; \
>  esac;
>
> Nice! The one thing I don't understand: why is "-" needed at the start
> (i.e. why do we need to ignore failure of this command?).

I don't recall exactly the reason Guix added it, I guess it was to allow
the build to "succeed" even if the man pages generation failed since
that doesn't impact the software to run.  However I am not sure if it's
a good idea.

> Combining the above with what I originally posted, I get:
>
> beetle.1: tbl_opts.h beetle$(EXEEXT)
> ## Exit gracefully if beetle.1 is not writeable, such as during distcheck!
> @if ( touch $@.w && rm -f $@.w; ) >/dev/null 2>&1; then \

IIUC this case silently ignores when the ‘beetle.1’ is not distributed,
which doesn't seem desirable for ‘make distcheck’, no?
 
> case '$?' in \
> *tbl_opts.h*) $(AM_V_P) && set -x || echo " HELP2MAN $@"; \
> $(top_srcdir)/build-aux/missing --run $(HELP2MAN) --no-info \
> --name="Forth virtual machine" \
> --output=$@ ./beetle$(EXEEXT);; \
> *) : ;; \
> esac; \
> fi

[1] https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html
[2] http://bootstrappable.org/

-- 
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

bug#31157: Advice for help2man does not work for parallel builds

[Reuben Thomas]

On 21 April 2018 at 16:13, Mathieu Lirzin  wrote:

> Hello Reuben,
>
> Reuben Thomas  writes:
>
> > In the manual, we are given the following pattern for using help2man
> > without breaking make distcheck:
> >
> > foo.1: foo.c $(top_srcdir)/configure.ac
> > $(MAKE) $(AM_MAKEFLAGS) foo$(EXEEXT)
> > help2man --output=foo.1 ./foo$(EXEEXT)
> >
> > The problem is that with make -j this can result in two attempts to
> > make a library in parallel (suppose that we have:
> >
> > foo_LDADD = libfoo.la
> > lib_LTLIBRARIES = libfoo.la
> >
> > ). This can fail, and in any case is wasteful.
>
> Have you identified the reason why this can fail?  because
>

​Because two independent parallel invocations of make ​can end up trying to
build the library (which is wasteful anyway) and some needed file can be
deleted by one invocation when the other is trying to use it to link the
library.

This is not ideal since this result in making ‘help2man’ (and ‘perl’
> transitively) a build dependency for tarball builders.
>

​I'm increasingly of the view this is not a problem. Perl is increasingly
reasonable as a build dep (it seems to be in most base systems now), and
help2man is small. Further, it's increasingly common to build from git
sources, or a tarball of git, or, as Debian does, to rerun
autoreconf/bootstrap at build time.​ However, for now I am being
conservative!

$(srcdir)/foo.1: foo.c foo$(EXEEXT)
> -@case '$?' in \
>   *foo.c*)
> ​​
> $(AM_V_P) && set -x || echo "  HELP2MAN $@"; \
>LANGUAGE= help2man --output="$(srcdir)/foo.1"
> ./foo$(EXEEXT);; \
>   *) : ;; \
> esac;
>

Nice! The one thing I don't understand: why is "-" needed at the start
(i.e. why do we need to ignore failure of this command?).

Combining the above with what I originally posted, I get:

beetle.1: tbl_opts.h beetle$(EXEEXT)
## Exit gracefully if beetle.1 is not writeable, such as during distcheck!
@if ( touch $@.w && rm -f $@.w; ) >/dev/null 2>&1; then \
case '$?' in \
  *tbl_opts.h*) $(AM_V_P) && set -x || echo "  HELP2MAN $@"; \
  $(top_srcdir)/build-aux/missing --run $(HELP2MAN) --no-info \
--name="Forth virtual machine" \
--output=$@ ./beetle$(EXEEXT);; \
  *) : ;; \
esac; \
fi

-- 
https://rrt.sc3d.org

bug#31157: Advice for help2man does not work for parallel builds

[Peter Johansson]

Hi Ruben and Mathieu,


On 4/22/2018 1:13 AM, Mathieu Lirzin wrote:

Hello Reuben,

Reuben Thomas  writes:


In the manual, we are given the following pattern for using help2man
without breaking make distcheck:

foo.1: foo.c $(top_srcdir)/configure.ac
$(MAKE) $(AM_MAKEFLAGS) foo$(EXEEXT)
help2man --output=foo.1 ./foo$(EXEEXT)

The problem is that with make -j this can result in two attempts to
make a library in parallel (suppose that we have:

foo_LDADD = libfoo.la
lib_LTLIBRARIES = libfoo.la

). This can fail, and in any case is wasteful.

Have you identified the reason why this can fail?  because


One problem is that the rule for foo.1 can be triggered before 
foo.$(EXEEXT) exists and the rule needs foo.$(EXEEXT) or help2man will 
fail. In a -j1 build this is never a problem as binaries are built 
before man pages.


When the 'missing' script changed behaviour in Automake 1.13 (and became 
useless imvho), we changed the rule in one project so foo.1 depended on 
foo.$(EXEEXT) but not if we are 1) building from a tarball and 2) 
foo.$(EXEEXT) exists.


http://dev.thep.lu.se/svndigest/browser/trunk/man/Makefile.am


Cheers,
Peter

bug#31157: Advice for help2man does not work for parallel builds

[Mathieu Lirzin]

Hello Reuben,

Reuben Thomas  writes:

> In the manual, we are given the following pattern for using help2man
> without breaking make distcheck:
>
> foo.1: foo.c $(top_srcdir)/configure.ac
> $(MAKE) $(AM_MAKEFLAGS) foo$(EXEEXT)
> help2man --output=foo.1 ./foo$(EXEEXT)
>
> The problem is that with make -j this can result in two attempts to
> make a library in parallel (suppose that we have:
>
> foo_LDADD = libfoo.la
> lib_LTLIBRARIES = libfoo.la
>
> ). This can fail, and in any case is wasteful.

Have you identified the reason why this can fail?  because 

> I'm using automake 1.15. I can't see anything since then that fixes this 
> problem.

Yes, this problem is likely to still be present in 1.16.1.

> The best workaround I could come up with was to revert the dependency to​
>
> ​ foo.1: foo$(EXEEXT)
>
> and then set distcleancheck_listfiles appropriately. Obviously, since
> this could hide other problems in the build system, it's not ideal.

This is not ideal since this result in making ‘help2man’ (and ‘perl’
transitively) a build dependency for tarball builders.

> Am I missing a better solution? If so, it should be added to the
> manual. If not, this problem should probably be documented. I'm
> finding that parallel make is becoming a must-have rather than a nice
> boost, given the proliferation of slow multi-core machines (for
> example: -j makes it feasible to hack on my phone; without it, builds
> are painfully slow).

I agree that it is important to provide a solution that support parallel
builds.  I remember that the issue has been adressed by Guix with
something like this (untested):

--8<---cut here---start->8---
$(srcdir)/foo.1: foo.c foo$(EXEEXT)
-@case '$?' in \
  *foo.c*) $(AM_V_P) && set -x || echo "  HELP2MAN $@"; \
   LANGUAGE= help2man --output="$(srcdir)/foo.1" 
./foo$(EXEEXT);; \
  *) : ;; \
esac;
--8<---cut here---end--->8---

This solution handles silent rules and possible localization of the
‘./foo --help’ output.  However its limitation is that if you have
another rule with ‘$(srcdir)/foo.1’ as a prerequisite then it will be
triggered at every build.

WDYT?

Thanks for the report.

-- 
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

bug#31157: Advice for help2man does not work for parallel builds

[Reuben Thomas]

In the manual, we are given the following pattern for using help2man
without breaking make distcheck:

 foo.1: foo.c $(top_srcdir)/configure.ac
 $(MAKE) $(AM_MAKEFLAGS) foo$(EXEEXT)
 help2man --output=foo.1 ./foo$(EXEEXT)

The problem is that with make -j this can result in two attempts to make a
library in parallel (suppose that we have:

foo_LDADD = libfoo.la
lib_LTLIBRARIES = libfoo.la

). This can fail, and in any case is wasteful.

I'm using automake 1.15. I can't see anything since then that fixes this
problem.

The best workaround I could come up with was to revert the dependency to​

​ foo.1: foo$(EXEEXT)

and then set distcleancheck_listfiles appropriately. Obviously, since this
could hide other problems in the build system, it's not ideal.

Am I missing a better solution? If so, it should be added to the manual. If
not, this problem should probably be documented. I'm finding that parallel
make is becoming a must-have rather than a nice boost, given the
proliferation of slow multi-core machines (for example: -j makes it
feasible to hack on my phone; without it, builds are painfully slow).

-- 
https://rrt.sc3d.org