Re: improved timestamp resolution test (was: 1.16.90 regression: configure now takes 7 seconds to start)

[Jacob Bachmeyer]

Karl Berry wrote:
Does BSD ls(1) support "--time=ctime --time-style=full-iso"?  


BSD ls does not support any --longopts. Looking at the man page,
I don't see "millisecond" or "subsecond" etc. mentioned, though I could
easily be missing it. E.g.,
  https://man.freebsd.org/cgi/man.cgi?ls

Even if there is such an option, I am skeptical of how portable it would
be, or trying to discern whether it is really working or not. All the
evidence so far is that it is very difficult to determine whether
subsecond mtimes are sufficiently supported or not. Speaking in general,
I don't think trying to get into system-specific behaviors, of whatever
kind, is going to help.


[*sigh*]

It seems that there is no good way for configure to read timestamps, so 
we are limited to testing if file ages are distinguishable.


Still, could we use make(1) for *all* of the testing and not use `ls -t` 
at all?  A rough outline would be something like:  (lightly tested; runs 
in about 2.2s here)


8<--
# The case below depends on the 1/10 + 9/10 = 10/10 pattern.
am_try_resolutions="0.01 0.09 0.1 0.9 1"
echo '#' > conftest.mk
i=0
for am_try_res in $am_try_resolutions; do
 echo ts${i} > conftest.ts${i}
 sleep $am_try_res
 echo "conftest.ts${i}: conftest.ts"`expr 1 + $i` >> conftest.mk
 echo "echo $am_try_res" >> conftest.mk
 i=`expr 1 + $i`
done
echo end > conftest.ts${i}
# This guess can be one step too fast, if the shorter delay just
#  happened to span a clock tick boundary.
am_resolution_guess=`make -f conftest.mk conftest.ts0 | tail -1`
case $am_resolution_guess in
 *9)
   i=no
   for am_try_res in $am_try_resolutions; do
 if test x$i = xyes; then
am_resolution=$am_try_res
break
 fi
 test x$am_try_res = x$am_resolution_guess && i=yes
   done
 ;;
 *)
   am_resolution=$am_resolution_guess
 ;;
esac
8<--


The trick is that the various options form a dependency chain, but the 
command make will execute does /not/ actually touch the target, so it 
stops when the files are no longer distinguishable.  This distinguishes 
between a tmpfs (which has nanosecond resolution here) and /home (which 
is an older filesystem with only 1-second resolution).  I am not sure 
what it does with FAT yet.  There should be some way to use 0.1+0.9+1 = 
2 and 0.01+0.09+0.1+0.9+1 > 2 to check for that (accurately!) without 
further sleeps.



-- Jacob

Re: 1.16.90 regression: configure now takes 7 seconds to start

[Jacob Bachmeyer]

dherr...@tentpost.com wrote:
At some point, it becomes unreasonable to burden common platforms with 
delays that only support relatively obscure and obsolete platforms.  
Configure scripts already have a bad reputation for wasting time.  
Even if they are faster than editing a custom makefile, they are idle 
instead of active time for the user, so waiting is harder.


I feel that 6-second test delays or 2-second incremental delays later 
qualify as clearly unreasonable.  The 1-second timestamps are 
borderline unreasonable.  Cross-compiling with a decent filesystem is 
more reasonable.


One second timestamp granularity is classic POSIX, and apparently also 
modern NetBSD.  We must support it.


Why can't we resolve this by requiring systems with 2-second 
resolution to set a flag in config.site?  That moves the burden closer 
to where it belongs.


First, because configure scripts are supposed to Just Work without 
particular expertise on the part of the user.  (Users with such 
deficient systems are least likely to have the expertise to handle 
that.)  Second, because timestamp resolution is actually per-volume, 
which in the POSIX model, means it varies by directory.  You can even 
have a modern filesystem (with nanosecond granularity) mounted on a 
directory in a FAT filesystem (with two second granularity) and 
ultimately a root filesystem with one second granularity.


In fact, the machine on which I type this has all three:  any tmpfs has 
nanosecond resolution, but /home has been carried for many years since 
mkfs and has one-second resolution, and I have removable media that is 
formatted FAT with its infamous two-second resolution.  All of these, 
when in use, appear in the same hierarchical filesystem namespace.



-- Jacob

Re: Bug Resilience Program of German Sovereign Tech Fund

[Jacob Bachmeyer]

Karl Berry wrote:

[...]

 > and reduce technical debt.

I don't know what that means. I instinctively shy away from such
vague buzzwords.
  


Essentially, "technical debt" means "stuff on the TODO list" and more 
specifically the accumulation of "good enough for now; fix it later" 
that tends to happen in software projects.



As for "modernizing" autoconf/make, mentioned in other msgs, that's the
last thing that should be done. We go to a lot of trouble to make the
tools work on old systems that no one else supports. For example, I can
just picture them saying "oh yes, you should use $(...) instead of
`...`" and other such "modern" shell constructs. Or "use Perl module
xyz to simplify", where xyz only became available a few years ago. Etc.
  


If you make them run their patches past the mailing list, I will happily 
complain if they try to break backwards compatibility without a very 
good reason.  Remember Time::HiRes and perl 5.6?  :-)



-- Jacob

Re: 1.16.90 regression: configure now takes 7 seconds to start

[Jacob Bachmeyer]

Karl Berry wrote:

bh> Seen e.g. on NetBSD 10.0.

Which doesn't support subsecond mtimes?

jb> Maybe the best answer is to test for subsecond timestamp
granularity first, and then only do the slow test to distinguish
between 1-second and 2-second granularity if the subsecond
granularity test gives a negative result?

Unfortunately, that is already the case. The function
(_AM_FILESYSTEM_TIMESTAMP_RESOLUTION in m4/sanity.m4) does the tests
starting with .01, then .1, then 1. Searching for sleep [012] in Bruno's
log confirms this.
  


So we are hitting the one-second timestamp granularity path because 
there is a modern system that does not have sub-second timestamp 
granularity, and that path is annoyingly slow.



If I understand correctly, Bruno's goal is to omit the "1" test if we
can detect that we're not on a fat filesystem. But I admit I don't like
trying to inspect filesystem types. That way lies madness, it seems to
me, and this whole function is already maddening enough. E.g., mount
and/or df could hang if NFS is involved.
  


I agree, although I had not considered the possibilities of problems 
with NFS.



It seems to me that using stat doesn't help because it's not available
by default on the bsds etc., as Bruno pointed out.
  


Does BSD ls(1) support "--time=ctime --time-style=full-iso"?  That would 
give equivalent information as stat(1) and, if at least one file has an 
odd seconds field, would rule out FAT quickly.  It could also indicate 
one-second granularity, if all subsecond parts are zero.  The slow test 
would still be required to confirm the worst case:  all timestamps are 
even because the filesystem has 2-second timestamp granularity.



The simple change is to omit the make test if we are at resolution 1.
That will save 4 seconds. Omitting it is justified because the make test
is only there for the sake of makes that are broken wrt subsecond
timestamps. I will do that.
  


If the critical issue is whether or not make(1) correctly handles 
subsecond timestamp granularity, why not simply test if make(1) 
recognizes subsecond timestamp differences and remove the other tests?  
If we are on a filesystem that does not have subsecond timestamp 
granularity, make will not have it either.



That will leave 2+ sec of sleeping, but if we are to reliably support
fat, I don't see a good alternative. At least it's not as bad as 6+.
Any other ideas?


As I hinted at, could we move the entire test into make(1) somehow?  
Could we lay out a set of files with timestamps differing by .01, .1, 1 
seconds and then see which pairs have ages distinguishable by make(1)?  
That should complete in less than 2 seconds:  1.11 seconds to make the 
files and less than half a second to run make(1).  If we can find a 
portable way to read timestamps to 1-second resolution, we can confirm 
not being on FAT---there will be one "odd file out" and the others will 
all have either odd or even timestamps.  If the timestamps all match, we 
can assume 2-second granularity without further testing, or do the slow 
test to confirm it.  If the "odd file out" has a timestamp two seconds 
ahead of the others, we *know* the filesystem has 2-second granularity 
and we crossed a "tick" boundary while making the files.


Alternately, could we improve the UI by emitting one additional dot per 
approximate second during the test?  Reassure the user that, yes, 
configure is doing something, even if all we can actually do is wait for 
the clock to advance.



-- Jacob

Re: 1.16.90 regression: configure now takes 7 seconds to start

[Jacob Bachmeyer]

Bruno Haible wrote:

Hi Jacob,

  

AFAIU, the 4x sleep 0.1 are to determine whether
am_cv_filesystem_timestamp_resolution should be set to 0.1 or to 1.
OK, so be it.

But the 6x sleep 1 are to determine whether
am_cv_filesystem_timestamp_resolution should be set to 1 or 2.
2 is known to be the case only for FAT/VFAT file systems. Therefore
here is a proposed patch to speed this up. On NetBSD, it reduces
the execution time of the test from ca. 7 seconds to ca. 0.5 seconds.
  
The problem with the proposed patch is that it tries to read a 
filesystem name instead of testing for the feature.  This would not be 
portable to new systems that use a different name for their FAT 
filesystem driver.



I can amend the patch so that it uses `uname -s` first, and does the
optimization only for the known systems (Linux, macOS, FreeBSD, NetBSD,
OpenBSD, Solaris).
  


This still has the same philosophical problem:  testing for a known 
system rather than for the feature we actually care about.  (We could 
also identify FAT with fair confidence by attempting to create a file 
with a name containing a character not allowed on the FAT filesystem, 
but I remember Linux having had at least one extended FAT driver 
("umsdos" if I remember correctly) that lifted the name limits, but I do 
not remember if it also provided improved timestamps.)


I think the test can be better optimized for the common case by first 
checking if stat(1) from GNU coreutils is available ([[case `stat 
--version` in *coreutils*) YES;; *) NO;; esac]])



Sure, if GNU coreutils 'stat -f' is available, things would be easy.
But typically, from macOS to Solaris, it isn't.

You can't achieve portability by using a highly unportable program
like 'stat'. That's why my patch only uses 'df' and 'mount'.
  


You can use anything in configure, *if* you first test for it and have a 
fallback if it is not available.  In this case, I am proposing testing 
for 'stat -f', using it to examine conveniently-available timestamps to 
establish an upper bound on timestamp granularity if we can, and falling 
back to the current (slow) tests if not.  Users of the GNU system will 
definitely get the fast path.


and, if it is (common 
case and definitely so on the GNU system), checking [[case `stat 
--format=%y .` in *:??.0) SUBSEC_RESOLUTION=no;; *) 
SUBSEC_RESOLUTION=yes;; esac]] to determine if sub-second timestamps are 
likely to be available



I don't care much about the 0.4 seconds spent on determining sub-second
resolution. It's the 6 seconds that bug me.
  


If 'stat -f' is available, we should be able to cut that to 
milliseconds.  GNU systems will have 'stat -f', others might.  The slow 
path would remain available if the fast path cannot be used.  Using a 
direct feature test for 'stat -f' might motivate the *BSDs to also 
support it.


To handle filesystems with 2-second timestamp resolution, check the 
timestamp on configure, and arrange for autoconf to ensure that the 
timestamp of a generated configure script is always odd



Since a tarball can be created on ext4 and unpacked on vfat FS,
  

That is exactly the situation I am anticipating here.

this would mean that autoconf needs to introduce a sleep() of up to
1 second, _regardless_ on which FS it is running. No, thank you,
that is not a good cure to the problem.


One second, once, when building configure, to ensure that configure will 
have an odd timestamp... does autoconf normally complete in less than 
one second?  Would this actually increase the running time 
significantly?  Or, as Simon Richter mentioned, use the utime builtin 
(Autoconf is now written in Perl) to advance the mtime of the created 
file by one second before returning with no actual delay.


The bigger problem would be that it would be impossible to properly 
package such a configure script if using a filesystem with 2-second 
granularity.  Such a configure script would always be unpacked with an 
even timestamp (because it was packaged with an even timestamp) and the 
2-second granularity test would give a false positive if the filesystem 
actually has 1-second granularity, but configure itself was generated on 
a 2-second granularity filesytem.  The suggested tests for sub-second 
granularity would still work correctly on the unpacked files, 
however---if you can see non-zero fractional seconds in timestamps, you 
know that you are not on a 2-second granularity filesystem.


Maybe the best answer is to test for subsecond timestamp granularity 
first, and then only do the slow test to distinguish between 1-second 
and 2-second granularity if the subsecond granularity test gives a 
negative result?  Most modern systems will have the subsecond timestamp 
granularity, so would need only the 0.4 second test; older systems would 
need the full 6.4 second test, but would still work reliably.  At worst, 
we might need to extend the 0.4 second test to 0.5 seconds, to confirm 
that we did not just happen to 

Re: 1.16.90 regression: configure now takes 7 seconds to start

[Jacob Bachmeyer]

Bruno Haible wrote:

[I'm writing to automake@gnu.org because bug-autom...@gnu.org
appears to be equivalent to /dev/null: no echo in
https://lists.gnu.org/archive/html/bug-automake/2024-06/threads.html
nor in https://debbugs.gnu.org/cgi/pkgreport.cgi?package=automake,
even after several hours.]

In configure scripts generated by Autoconf 2.72 and Automake 1.16.90,
one of the early tests
  checking filesystem timestamp resolution...
takes 7 seconds! Seen e.g. on NetBSD 10.0.

Logging the execution time, via
  sh -x ./configure 2>&1 | gawk '{ print strftime("%H:%M:%S"), $0; fflush(); }' 
> log1
I get the attached output. There are
  6x sleep 1
  4x sleep 0.1
That is, 6.4 seconds are wasted in sleeps. IBM software may do this;
but GNU software shouldn't.

AFAIU, the 4x sleep 0.1 are to determine whether
am_cv_filesystem_timestamp_resolution should be set to 0.1 or to 1.
OK, so be it.

But the 6x sleep 1 are to determine whether
am_cv_filesystem_timestamp_resolution should be set to 1 or 2.
2 is known to be the case only for FAT/VFAT file systems. Therefore
here is a proposed patch to speed this up. On NetBSD, it reduces
the execution time of the test from ca. 7 seconds to ca. 0.5 seconds.


The problem with the proposed patch is that it tries to read a 
filesystem name instead of testing for the feature.  This would not be 
portable to new systems that use a different name for their FAT 
filesystem driver.


I think the test can be better optimized for the common case by first 
checking if stat(1) from GNU coreutils is available ([[case `stat 
--version` in *coreutils*) YES;; *) NO;; esac]]) and, if it is (common 
case and definitely so on the GNU system), checking [[case `stat 
--format=%y .` in *:??.0) SUBSEC_RESOLUTION=no;; *) 
SUBSEC_RESOLUTION=yes;; esac]] to determine if sub-second timestamps are 
likely to be available; this has a 1-in-actual-ticks-per-second of 
giving a false negative.  These checks would be very fast, so could also 
be repeated with the access and inode change timestamps and/or extended 
to other files (`stat *`) for better certainty.  The basic concept 
should be sound, although the pattern matching used in the examples is a 
first cut.


The essential idea is that the fractional part beyond what the 
filesystem actually records will always read as zero, and unpacking an 
archive is not instant, so we should see every implemented fractional 
bit set at least once across files in the tree containing configure.


To handle filesystems with 2-second timestamp resolution, check the 
timestamp on configure, and arrange for autoconf to ensure that the 
timestamp of a generated configure script is always odd---that 
least-significant bit will be dropped when the script is unpacked on a 
filesystem with 2-second timestamp resolution.


If stat from GNU coreutils is not available, fall back to the current 
sleep(1)-based test and just eat the delay in the name of portability.  
The test checks only for "coreutils" because very old versions did not 
say GNU.  A better, functional test for stat(1) is probably also possible.



-- Jacob

Re: follow-up on backdoor CPU usage (was: libsystemd dependencies)

[Jacob Bachmeyer]

Jacob Bachmeyer wrote:
[...]  The preliminary reports that it was an RCE backdoor that would 
pass commands smuggled in public key material in SSH certificates to 
system(3) (as root of course, since that is sshd's context at that 
stage) are inconsistent with the slowdown that caused the backdoor to 
be discovered.  I doubt that SSH logins were using that code path, and 
the SSH scanning botnets almost certainly are not presenting 
certificates, yet it apparently (reports have been unclear on this 
point) was the botnet scanning traffic that led to the discovery of 
sshd wasting considerable CPU time in liblzma...


I am waiting for the proverbial other shoe to drop on that one.


I have been given 
(https://www.openwall.com/lists/oss-security/2024/04/18/1>) a 
satisfactory explanation for the inconsistency:  OpenSSH sshd uses 
exec(2) to reshuffle ASLR before accepting each connection, and the 
backdoor blob's tampering with the dynamic linking process greatly 
reduces the efficiency of ld.so on top of its own processing.  The 
observable wasted CPU time was the backdoor's excessively-complex 
initialization, rather than any direct effect on sshd connection processing.



-- Jacob

Re: GCC reporting piped input as a security feature

[Jacob Bachmeyer]

Zack Weinberg wrote:

On Tue, Apr 9, 2024, at 11:35 PM, Jacob Bachmeyer wrote:
  

Jan Engelhardt wrote:


On Tuesday 2024-04-09 05:37, Jacob Bachmeyer wrote:

  

In principle it could be posible to output something different to
describe this stramge situation explicitly.  For instance, output
"via stdin" as a comment, or output `stdin/../filename' as the file
name. (Programs that optimize the file name by deleting XXX/.../
are likely not to check whether XXX is a real directory.)
  

...

How about `/dev/stdin/-` if no filename has been specified with #line or
whatever, and `/dev/stdin/[filename]` if one has, where [filename] is
the specified filename with all leading dots and slashes stripped,
falling back to `-` if empty? /dev/stdin can be relied on to either not
exist or not be a directory, so these shouldn't ever be openable.


I like that idea, but would suggest expanding on it as 
"/dev/stdin/[working directory]//-" or "/dev/stdin/[working 
directory]//[full specified filename]".  The double slash allows tools 
that care to parse out the specified filename, while the working 
directory preceding it provides a hint where to find that file if the 
specified filename is relative, but the kernel will collapse it to a 
single slash if a tool just passes the "[working directory]//[specified 
filename]" to open(2).  Since the working directory should itself be an 
absolute name, there would typically be a double slash after the 
"/dev/stdin" prefix.  Something like 
"/dev/stdin//var/cache/build/foopkg-1.0.0///usr/src/foopkg-1.0.0/special.c.m4" 
as an artificial example.



-- Jacob

Re: GCC reporting piped input as a security feature

[Jacob Bachmeyer]

Jan Engelhardt wrote:

On Tuesday 2024-04-09 05:37, Jacob Bachmeyer wrote:
  

In principle it could be posible to output something different to
describe this stramge situation explicitly.  For instance, output "via
stdin" as a comment, or output `stdin/../filename' as the file name.
(Programs that optimize the file name by deleting XXX/.../ are likely
not to check whether XXX is a real directory.)
  

With the small difference that I believe the special marker should be ''
(with the angle brackets, as it is now), this could be another good idea.
Example output:  "[working directory][specified filename]" or
"[specified filename]///<>/[working directory]/".  GDB could be modified
[...]



This will likely backfire. Assuming you have a userspace program
which does not care about any particular substring being present, the
fullpath is passed as-is to the OS kernel, which *will* resolve it
component by component, and in doing so, stumble over the XXX/ part.
  


And upon so stumbling, return ENOENT or ENOTDIR.  Where is the harm 
there?  Input read from a pipe does not exist in the filesystem.



Better introduce a new DW_AT_ field for a stdin flag.
  


That would mean that older tools could be confused.  How about a new 
field for "source-specified filename" when that differs from the actual 
file being read?  That way, existing tools would still see "[working 
directory]/" and avoid confusion, which could be a security risk 
here.



-- Jacob

Re: GCC reporting piped input as a security feature

[Jacob Bachmeyer]

Alan D. Salewski wrote:

On 2024-04-08 22:37:50, Jacob Bachmeyer  spake thus:

Richard Stallman wrote:

[...]

In principle it could be posible to output something different to
describe this stramge situation explicitly.  For instance, output "via
stdin" as a comment, or output `stdin/../filename' as the file name.
(Programs that optimize the file name by deleting XXX/.../ are likely
not to check whether XXX is a real directory.)



With the small difference that I believe the special marker should be
'' (with the angle brackets, as it is now), this could be another
good idea.  Example output:  "[working directory][specified
filename]" or "[specified filename]///<>/[working directory]/".
GDB could be modified to recognize either form and read the specified
file (presumably some form of augmented C) but report that the sources
were transformed prior to compilation.  The use of triple-slash ensures
that these combined strings cannot be confused with valid POSIX
filenames, although I suspect that uses of these strings would have to
be a GNU extension to the debugging info format.


I do not think that the use of triple-slash (or any-N-slash) would
entirely prevent potential confusion with valid POSIX filenames, as
POSIX treats multiple slashes as equivalent to a single slash
(except in at the beginning of a path, where two slash characters
may have a different, implementation-defined meaning).

Since a pathname component name can basically contains any bytes
except  and , any token value chosen will likely have
some non-zero potential for confusion with a valid POSIX pathname.


Yes, this is the downside of the extreme flexibility of POSIX filename 
semantics.  Any C string is potentially a valid filename.



From SUSv4 2018[0] (update from 2020-04-30, which is what I happen
to have handy):


3.271 Pathname

A string that is used to identify a file. In the context of
POSIX.1-2017, a pathname may be limited to {PATH_MAX} bytes,
including the terminating null byte. It has optional beginning
 characters, followed by zero or more filenames separated
by  characters. A pathname can optionally contain one or
more trailing  characters. Multiple successive 
characters are considered to be the same as one , except
for the case of exactly two leading  characters.



Rats, I had forgotten that detail.  Emacs treats typing a second slash 
as effectively invalidating everything to the left, I remembered that 
some systems (and many URL schemes) use double-slash to indicate a 
network host, and I expected that 3 slashes would mean starting over at 
the root if that were ever presented to the kernel's filename resolution 
service.


On the other hand, we could use multiple slashes as a delimiter if GCC 
normalizes such sequences in input filename strings to single slash, 
which POSIX allows, according to the quote above.


The simplest solution would be to simply document and preserve the 
current behavior, which appears to be ignoring directives and recording 
the working directory and "" in the case of reading from a pipe, 
and making sure that no normal build procedure for any GNU package pipes 
source into the compiler.



-- Jacob

Re: GCC reporting piped input as a security feature

[Jacob Bachmeyer]

Richard Stallman wrote:

[[[ To any NSA and FBI agents reading my email: please consider]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > While it does not /prevent/ cracks, there is something we can ensure 
  > that we *keep* doing:  GCC, when reading from a pipe, records the input 
  > file as "" in debug info *even* if a "#" directive to set the 
  > filename has been included.  This was noticed by Adrien Nader (who 
  > posted it to oss-security; 
  > https://www.openwall.com/lists/oss-security/2024/04/03/2> and 
  > https://marc.info/?l=oss-security=171214932201156=2>; those are 
  > the same post at different public archives) and should provide a 
  > "smoking gun" test to detect this type of backdoor dropping technique in 
  > the future.  This GCC behavior should be documented as a security 
  > feature, because most program sources are not read from pipes.


Are you suggesting fixing GCC to put the specified file into those
linenumbers, or are you suggesting we keep this behavior
to help with analysis?
  


I am suggesting that we keep this behavior (and document it as an 
explicit security feature) to help with detection of any future similar 
cracks, and add provisions to the GNU Coding Standards to avoid false 
positives by requiring generated sources to appear in the filesystem 
instead of being piped to the compiler.



In principle it could be posible to output something different to
describe this stramge situation explicitly.  For instance, output "via
stdin" as a comment, or output `stdin/../filename' as the file name.
(Programs that optimize the file name by deleting XXX/.../ are likely
not to check whether XXX is a real directory.)
  


With the small difference that I believe the special marker should be 
'' (with the angle brackets, as it is now), this could be another 
good idea.  Example output:  "[working directory][specified 
filename]" or "[specified filename]///<>/[working directory]/".  
GDB could be modified to recognize either form and read the specified 
file (presumably some form of augmented C) but report that the sources 
were transformed prior to compilation.  The use of triple-slash ensures 
that these combined strings cannot be confused with valid POSIX 
filenames, although I suspect that uses of these strings would have to 
be a GNU extension to the debugging info format.  (If GNU-extended 
debugging information is inhibited, I think it is more important to 
declare that the input came from a pipe than to carry the specified 
filename.)  This might actually be a good idea in general if a directive 
specifies a filename with the same suffix but not the file being read.


As another layer against similar attacks, distribution packaging tools 
could grep the debug symbols for '' and raise alarms if matches 
are found.  Forbidding piping source to the compiler in the GNU Coding 
Standards would eliminate false positives.



-- Jacob

Re: detecting modified m4 files (was: GNU Coding Standards, automake, and the recent xz-utils backdoor)

[Jacob Bachmeyer]

Bruno Haible wrote:

Richard Stallman commented on Jacob Bachmeyer's idea:
  
  > > Another related check that /would/ have caught this attempt would be 
  > > comparing the aclocal m4 files in a release against their (meta)upstream 
  > > sources before building a package.  This is something distribution 
  > > maintainers could do without cooperation from upstream.  If 
  > > m4/build-to-host.m4 had been recognized as coming from gnulib and 
  > > compared to the copy in gnulib, the nonempty diff would have been 
  > > suspicious.


I have a hunch that some effort is needed to do that comparison, but
that it is feasible to write a script to do it could make it easy.
Is that so?



Yes, the technical side of such a comparison is relatively easy to
implement:
  - There are less than about 2000 or 5000 *.m4 files that are shared
between projects. Downloading and storing all historical versions
of these files will take ca. 0.1 to 1 GB.
  - They would be stored in a content-based index, i.e. indexed by
sha256 hash code.
  - A distribution could then quickly test whether a *.m4 file found
in a distrib tarball is "known".

The recurrently time-consuming part is, whenever an "unknown" *.m4 file
appears, to
  - manually review it,
  - update the list of upstream git repositories (e.g. when a project
has been forked) or the list of releases to consider (e.g. snapshots
of GNU Autoconf or GNU libtool, or distribution-specific modifications).

I agree with Jacob that a distro can put this in place, without needing
to bother upstream developers.


I have since thought of a simple solution that /would/ have caught this 
backdoor campaign in its tracks:  an "autopoint --check" command that 
simply compares the m4/ files (and possibly others?) that autopoint 
would copy in if m4/ were empty against the files that would be copied 
and reports any differences.  A newer serial in the package tree than 
the system m4 library produces a minor complaint; a file with the same 
serial and different contents produces a major complaint.  An older 
serial in the package tree should be reported, but is likely to be of no 
consequence if a distribution's packaging routine will copy in the 
known-good newer version before rebuilding configure.  Any m4/ files 
local to the package are simply reported, but those are also in the 
package's Git repository.


Distribution package maintainers would run "autopoint --check" and pass 
any suspicious files to upstream maintainers for evaluation.  (The 
distribution's own packaging system can trace an m4 file in the system 
library came to its upstream package.)  The modified build-to-host.m4 
would have been very /unlikely/ to slip past the 
gnulib/gettext/Automake/Autoconf maintainers, although few distribution 
packagers would have had suspicions.  The gnulib maintainers would know 
that gl_BUIILD_TO_HOST should not be checking /anything/ itself and the 
crackers would have been caught.


This should be effective in closing off a large swath of possible 
attacks:  a backdoor concealed in binary test data (or documentation) 
requires some visible means to unpack it, which means the unpacker must 
appear in source somewhere.  While the average package maintainer might 
not be able to make sense of a novel m4 file, the maintainers of GNU's 
version of that file /will/ be able to recognize such chicanery, and the 
"red herrings" the cracker added for obfuscation would become a 
liability.  Without them, the effect of the new code is more obvious, so 
the crackers lose either way.



-- Jacob

Re: GCC reporting piped input as a security feature (was: GNU Coding Standards, automake, and the recent xz-utils backdoor)

[Jacob Bachmeyer]

Richard Stallman wrote:

[[[ To any NSA and FBI agents reading my email: please consider]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

[...]

When considering any such change, we still should consider the question:
will this actually prevent cracks, or will it rather give crackers
an additional way to check that their activities can't be detected.
  


While it does not /prevent/ cracks, there is something we can ensure 
that we *keep* doing:  GCC, when reading from a pipe, records the input 
file as "" in debug info *even* if a "#" directive to set the 
filename has been included.  This was noticed by Adrien Nader (who 
posted it to oss-security; 
https://www.openwall.com/lists/oss-security/2024/04/03/2> and 
https://marc.info/?l=oss-security=171214932201156=2>; those are 
the same post at different public archives) and should provide a 
"smoking gun" test to detect this type of backdoor dropping technique in 
the future.  This GCC behavior should be documented as a security 
feature, because most program sources are not read from pipes.


The xz backdoor dropper took great pains to minimize its use of the 
filesystem; only the binary blob ever touches the disk, and that 
presumably because there is no other way to feed it into the linker.  If 
debug info is regularly checked for symbols obtained from "" and 
the presence of such symbols reliably indicates funny business, then we 
force crackers to risk leaving more direct traces in the filesystem, 
instead of being able to patch the code "in memory" and feed an 
ephemeral stream to the compiler.  The "Jia Tan" crackers seem to have 
put a lot of work into minimizing the "footprint" of their dropper, so 
we can assume that this must have been important to them.


To avoid false positives if this test is used, we might want to add a 
rule to the GNU Coding Standards (probably in the "Makefile Conventions" 
section) that code generated with other utilities MUST always be 
materialized in the filesystem and MUST NOT be piped into the compiler.



-- Jacob

Re: compressed release distribution formats (was: GNU Coding Standards, automake, and the recent xz-utils backdoor)

[Jacob Bachmeyer]

Eric Blake wrote:

[adding in coreutils, for some history]

[...]

At any rate, it is now obvious (in hindsight) that zstd has a much
larger development team than xz, which may alter the ability of zstd
being backdoored in the same way that xz was, merely by social
engineering of a lone maintainer.
  


That just means that a cracker group needs to plant a mole in a larger 
team, which was effectively the goal of the sockpuppet campaign against 
the xz-utils maintainer, except that the cracker's sockpuppet was the 
second member of a two-member team.  I see no real difference here.


I would argue that GNU software should be consistently available in at 
least one format that can be unpacked using only tools that are also 
provided by the GNU project.  I believe that currently means "gzip", 
unfortunately.  We should probably look to adopt another one; perhaps 
the lzip maintainer might be interested?



It is also obvious that having GNU distributions available through
only a SINGLE compression format, when that format may be vulnerable,
  
The xz format is not vulnerable, or at least has not been shown to be so 
in the sense of security risks, and only xz-utils was backdoored.  Nor 
is there only one implementation:  7-zip can also handle xz files.

is a dis-service to users when it is not much harder to provide
tarballs in multiple formats.  Having multiple tarballs as the
recommendation can at least let us automate that each of the tarballs
has the same contents,
Agreed.  In fact, if multiple formats can be produced concurrently, we 
could validate that the compressed tarballs are actually identical.  
(Generate using `tar -cf - [...] | tee >(compress1 >[...].tar.comp1) | 
tee >(compress2 >[...].tar.comp2) | gzip -9 >[...].tar.gz` if you do not 
want to actually write the uncompressed tarball to the disk.)  But if 
tarlz is to be used to write the lzipped tarball, you probably want to 
settle for "same file contents", since tarlz only supports pax format 
and we may want to allow older tar programs to unpack GNU releases.

 although it won't make it any more obvious
whether those contents match what was in git (which was how the xz
backdoor got past so many people in the first place).
This is another widespread misunderstanding---almost all of the xz 
backdoor was hidden in plain sight (admittedly, compressed and/or 
encrypted) *in* the Git repository.  The only piece of the backdoor not 
found in Git was the modified build-to-host.m4.  The xz-utils project's 
standard practice had been to /not/ commit imported m4 files, but to 
bring them in when preparing release tarballs.  The cracker simply 
rolled the "key" to the dropper into the release tarball.  I still have 
not seen whether the configure script in the release tarball was built 
with the modified build-to-host.m4 or if the crackers were depending on 
distribution packagers to regenerate configure.


Again, everything present in both Git and the release tarball /was/ 
/identical/.  There were no mismatches, only files added to the release 
that are not in the repository, and that are /expected/ to be added to a 
release.



-- Jacob

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Richard Stallman wrote:

[[[ To any NSA and FBI agents reading my email: please consider]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > My first thought was that Autoconf is a relatively trivial attack vector 
  > since it is so complex and the syntax used for some parts (e.g. m4 and 
  > shell scripts) is so arcane.  In particular, it is common for Autotools 
  > stuff to be installed on a computer (e.g. by installing a package from 
  > an OS package manager) and then used while building.  For example, there 
  > are large collections of ".m4" files installed.  If one of the m4 files 
  > consumed has been modified, then the resulting configure script has been 
  > modified.


Can anyone think of a feasible way to prevent this sort of attack?
  


There have been some possibilities suggested on other branches of the 
discussion.  I have changed the subject of one of those to "checking 
aclocal m4 files" to highlight it.  There is progress being made, but 
the solutions appear to be outside the direct scope of the GNU build 
system packages.



Someone suggested that configure should not use m4 files that are
lying around, but rather should fetch them from standard release points,
WDYT of that idea?
  


Autoconf configure scripts do not use nearby m4 files and do not require 
m4 at all; aclocal collects the files in question into aclocal.m4 (I 
think) and then autoconf uses that (and other inputs) to /produce/ 
configure.  (This may seem like a trivial point, but exact derivations 
and their timing were critical to how the backdoor dropper worked.)  
Other tools (at least autopoint from GNU gettext, possibly others) are 
used to automatically scan a larger set of m4 files stored on the system 
and copy those needed into the m4/ directory of a package source tree, 
in a process conceptually similar to how the linker pulls only needed 
members from static libraries when building an executable.  All of this 
is done on the maintainer's machine, so that the finished configure 
script is included in the release tarball.


There have been past incidents where malicious code was directly added 
to autoconf-generated configure scripts, so (as I understand) 
distribution packagers often regenerate configure before building a 
package.  In /this/ case, the crackers (likely) modified the version of 
build-to-host.m4 on /their/ computer, so the modified file would be 
copied into the xz-utils/m4 directory in the release tarball and used 
when distribution packagers regenerate configure before building the 
package.


Fetching these files from standard release points would require an index 
of those standard release points, and packages are allowed to have their 
own package-specific macros as well.  The entire system dates from well 
before ubiquitous network connectivity could be assumed (anywhere---and 
that is still a bad assumption in the less prosperous parts of the 
world), so release tarballs are meant to be self-contained, including 
copies of "standard" macros needed for configure but not supplied by 
autoconf/automake/etc.



-- Jacob

Re: checking aclocal m4 files (was: GNU Coding Standards, automake, and the recent xz-utils backdoor)

[Jacob Bachmeyer]

Bruno Haible wrote:

Jacob Bachmeyer wrote:
  
Another related check that /would/ have caught this attempt would be 
comparing the aclocal m4 files in a release against their (meta)upstream 
sources before building a package.  This is something distribution 
maintainers could do without cooperation from upstream.  If 
m4/build-to-host.m4 had been recognized as coming from gnulib and 
compared to the copy in gnulib, the nonempty diff would have been 
suspicious.



True.

Note, however, that there would be some false positives:


True; all of these are Free Software, so a non-empty diff would still 
require manual review.



 libtool.m4 is often shipped modified,
  a) if the maintainer happens to use /usr/bin/libtoolize and
 is using a distro that has modified libtool.m4 (such as Gentoo), or
  


Distribution libtool patches could be accumulated into the set of "known 
sources".



  b) if the maintainer intentionally improved the support of specific
 platforms, such as Solaris 11.3.
  


In this case, the distribution maintainer should ideally take up pushing 
those improvements back to upstream libtool, if they are suitably general.



Also, for pkg.m4 there is no single upstream source. They distribute
a pkg.m4.in, from which pkg.m4 is generated on the developer's machine.
  


This would be a special case, but could be treated as a package-specific 
m4 file anyway, since the developer must generate it.  The developer 
could also write their own m4 macros to use with autoconf.



But for macros from Gnulib or the Autoconf macros archive, this is a
reasonable check to make.


This type of check could also allow "sweeping" improvements upstream, in 
the case of a package maintainer that may be unsure of how to upstream 
their changes.  (Of course, upstream needs to be careful about blindly 
collecting improvements, lest some of those "improvements" turn out to 
have come from cracker sockpuppets...)



-- Jacob

Re: binary data in source trees (was: GNU Coding Standards, automake, and the recent xz-utils backdoor)

[Jacob Bachmeyer]

Richard Stallman wrote:

[[[ To any NSA and FBI agents reading my email: please consider]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > The issue seems to be releases containing binary data for unit tests, 
  > instead of source or scripts to generate that data.  In this case, that 
  > binary data was used to smuggle in heavily obfuscated object code.


If this is the crucial point, we could put in the coding standards
(or the maintainers' guide) not to do this.


On another branch of this discussion, Zack Weinberg noted that binary 
test data may be unavoidable in some cases.  (A base64 blob or hex dump 
may as well be a binary blob.)  Further, manuals often contain images, 
some of which may be in binary formats, such as PNG.  To take this all 
the way, we would have to require that all documentation graphics be 
generated from readable sources.  I know TikZ exists but am unsure how 
well it could be integrated into Texinfo, for example.



-- Jacob

Re: reproducible dists and builds (was: GNU Coding Standards, automake, and the recent xz-utils backdoor)

[Jacob Bachmeyer]

Richard Stallman wrote:

[[[ To any NSA and FBI agents reading my email: please consider]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > What would be helpful is if `make dist' would guarantee to produce the same
  > tarball (bit-to-bit) each time it is run, assuming the tooling is the same
  > version.  Currently I believe that is not the case (at least due to 
timestamps)

Isn't this a description of "reproducible compilation"?
  


No, but it is closely related.  Compilation produces binary executables, 
while `make dist` produces a freestanding /source/ archive.



We want to make that standard, but progress is inevitably slow
because many packages need to be changed.


I am not actually sure that that is actually a good idea.  (Well, it is 
mostly a good idea except for one issue.)  If compilation is strictly 
deterministic, then everyone ends up with identical binaries, which 
means an exploit that cracks one will crack all.  Varied binaries make 
life harder for crackers developing exploits, and may even make "one 
exploit to crack them all" impossible.  This is one of the reasons that 
exploits have long hit Windows (where all the systems are identical) so 
much harder than the various GNU/Linux distributions (where the binaries 
are likely different even before distribution-specific patches are 
considered).


Ultimately, this probably means that we should have both an /ability/ 
for deterministic compilation and either a compiler mode or 
post-processing pass (a linker option?) to intentionally shuffle the 
final executable.



-- Jacob

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Eric Gallager wrote:

On Tue, Apr 2, 2024 at 12:04 AM Jacob Bachmeyer  wrote:
  

Russ Allbery wrote:


[...] I think one useful principle that's
emerged that doesn't disrupt the world *too* much is that the release
tarball should differ from the Git tag only in the form of added files.
  

 From what I understand, the xz backdoor would have passed this check.

[...]


[...] In other
words, even if a proposal wouldn't have stopped this particular
attack, I don't think that's a reason not to try it.


I agree that there may be dumber crackers who /would/ get caught by such 
a check, but I want to ensure that we do not end up thinking that we 
have a solution and the problem is solved and everyone is happy ... and 
then we get caught out when it happens again.


I should clarify also that I think that this proposal *is* a good idea, 
but we should remain aware that it would not have prevented this incident.


Maneuvering around back to topic, aclocal m4 files are fairly small, 
perhaps always carrying all of them that a package uses in the 
repository should be considered a good practice?  (In other words, 
autogen.sh should *not* run autopoint---the files autopoint adds should 
be in the repository.)  If such a practice were followed, that would 
have made checking for altered files between repository and release 
effective, or it would have forced the cracker to target the backdoor 
more widely and place the altered build-to-host.m4 in the repository, 
increasing the probability of discovery.


Wording that as a policy:  "All data inputs used to construct the build 
scripts for a package shall be stored in the package's repository."


Another related check that /would/ have caught this attempt would be 
comparing the aclocal m4 files in a release against their (meta)upstream 
sources before building a package.  This is something distribution 
maintainers could do without cooperation from upstream.  If 
m4/build-to-host.m4 had been recognized as coming from gnulib and 
compared to the copy in gnulib, the nonempty diff would have been 
suspicious.



-- Jacob

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Richard Stallman wrote:

[[[ To any NSA and FBI agents reading my email: please consider]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > `distcheck` target's prominence to recommend it in the "Standard
  > Targets for All Users" section of the GCS? 


  > Replying as an Automake developer, I have nothing against it in
  > principle, but it's clearly up to the GNU coding standards
  > maintainers. As far as I know, that's still rms (for anything
  > substantive)

To make a change in the coding standards calls for a clear and
specific proposal.  If people think a change is desirable, I suggest
making one or more such proposals.

Now for a bit of speculation.  I speculate that a cracker was careless
and failed to adjust certain details of a bogus tar ball to be fully
consistent, and that `make distcheck' enabled somene to notice those
errors.

I don't have any real info about whether that is so.  If my
speculation is mistaken, please say so.


I believe it is completely mistaken.  As I understand, the crocked 
tarballs would have passed `make distcheck` with flying colors.  The 
rest of your questions about it therefore have no answer.


On a side note, thanks for Emacs:  when I finally extracted a copy of 
the second shell script in the backdoor dropper, Emacs made short work 
(M-< M-> C-M-\) of properly indenting it and making the control flow 
obvious.  Misunderstandings of that control flow have been fairly 
common.  (I too had it wrong before I finally had a nicely indented copy.)


The backdoor was actually discovered in operation on machines running 
testing package versions.  It caused sshd to consume an inordinate 
amount of CPU time, with profiling reporting that sshd was spending most 
of its time in liblzma, a library not even linked in sshd.  (The "rogue" 
library had been loaded as a dependency of libsystemd, which the 
affected distributions had patched sshd to use for startup notification.)


I will send a more detailed reply on the other thread, since its subject 
is more appropriate.



-- Jacob

Re: role of GNU build system in recent xz-utils backdoor

[Jacob Bachmeyer]

Richard Stallman wrote:

[[[ To any NSA and FBI agents reading my email: please consider]]]
[[[ whether defending the US Constitution against all enemies, ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I was recently reading about the backdoor announced in xz-utils the
  > other day, and one of the things that caught my attention was how
  > (ab)use of the GNU build system played a role in allowing the backdoor
  > to go unnoticed: https://openwall.com/lists/oss-security/2024/03/29/4

[...]

I don't want to get involved in fixing the bug, but I want to
make sure the GNU Project is working on it.
  


I believe the GNU Project is blameless and uninvolved in this matter.  I 
am aware of possible elements used in the attack from the GNU Project, 
however two of them are innocent improvements abused by the cracker and 
the third was modified by the cracker.


The backdoor is in two major segments:  a binary blob hidden in a 
testsuite data file and two shell scripts that drop it, also hidden in 
testsuite data files.  A modified version of build-to-host.m4 from 
gnulib is used to insert code into configure to initially extract the 
dropper and run it (via pipeline---the dropper shell scripts never touch 
the filesystem).


If several conditions are met (building a shared library on 
'x86_64-*-linux-gnu', HAVE_FUNC_ATTRIBUTE_IFUNC, using the GNU toolchain 
(or at least a linker that claims to be "GNU ld" and a compiler invoked 
as "gcc"), and building under either dpkg or rpm), the dropper extracts 
a binary blob and links it with a legitimate object, which is patched 
and recompiled (using sed in a pipeline; the modified C source never 
touches the filesystem) to call a function exported from the blob.


The aclocal m4 files involved in the attack were never committed to the 
xz Git repository, instead being added to each release tarball using 
autopoint.  (This was the package's standard practice before the 
attack.)  The offending build-to-host.m4 was modified by the cracker, 
either directly in the release tree or at the location where autopoint 
will find it.  Some of the modifications "sound like" the cracker may 
have used a language model trained on other GNU sources---they are very 
plausible at first glance.


The elements from the GNU Project potentially implicated are 
build-to-host.m4, gettext.m4, and the ifunc feature in glibc.  All of 
these turn out to be innocent.


The initial "key" that activated the backdoor dropper was a modified 
version of the gl_BUILD_TO_HOST macro from gnulib.  The dropper also 
checks m4/gettext.m4 for the text "dnl Convert it to C string syntax." 
and fails to extract the blob if found.  It turns out that 
gl_BUILD_TO_HOST is used only as a dependency of gettext.m4 and that 
that comment was removed in the same commit that factored out 
gl_BUILD_TO_HOST to gnulib.  (commit 
3adaddd73c8edcceaed059e859bd5262df65fc5a in GNU gettext repository is by 
Bruno Haible; his involvement in the backdoor campaign is *extremely* 
unlikely in my view)


The "ifunc" feature merely allows the programmer to store function 
pointers in the PLT instead of the data segment when alternate 
implementations of a function are involved.  Theoretically, it should 
actually be a security improvement, as the PLT can be made read-only 
after all links are resolved, while the data segment must remain 
generally writable.


The backdoor will not be dropped if the use of ifunc is disabled or if 
the feature is unavailable.  I currently believe that the cracker used 
ifunc support as a covert flag to disable the backdoor when the oss-fuzz 
project was scanning the package.  I also suspect that the cracker's 
claim that ifuncs cause segfaults under -fsanitize=address (in the 
message for commit ee44863ae88e377a5df10db007ba9bfadde3d314 in the xz 
Git repository) may have been less than honest; that commit also gives 
credit to another of the cracker's sockpuppets for the original patch 
and was committed by the cracker's main "Jia Tan" sockpuppet, so the 
involvement of the primary maintainer (who is listed as the author of 
the commit in Git) is uncertain.  (In other words, the xz Git repository 
likely contains blatant lies put there by the cracker.)


Looking into this a little more, I now know what the dropper's C source 
patch does:  the blob's initialization entrypoint is named _get_cpuid 
(note only one leading underscore) and is called from an inserted static 
inline function that crc{32,64}_resolve (the ifunc resolvers that choose 
CRC implementations) are patched to call.  The dropper also ensures (by 
modifying liblzma_la_LDFLAGS in src/liblzma/Makefile) that liblzma.so 
will be linked with -Wl,-z,now so that ifuncs are resolved as the shared 
object is loaded.  That is how the backdoor blob initially gains control 
at a time during early process initialization when the PLT is still 
writable despite other hardening.



-- Jacob

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Zack Weinberg wrote:

On Mon, Apr 1, 2024, at 2:04 PM, Russ Allbery wrote:
  

"Zack Weinberg"  writes:


It might indeed be worth thinking about ways to minimize the
difference between the tarball "make dist" produces and the tarball
"git archive" produces, starting from the same clean git checkout,
and also ways to identify and audit those differences.
  

There is extensive ongoing discussion of this on debian-devel. There's
no real consensus in that discussion, but I think one useful principle
that's emerged that doesn't disrupt the world *too* much is that the
release tarball should differ from the Git tag only in the form of
added files. Any files that are present in both Git and in the release
tarball should be byte-for-byte identical.



That dovetails nicely with something I was thinking about myself.
Obviously the result of "make dist" should be reproducible except for
signatures; to the extent it isn't already, those are bugs in automake.
But also, what if "make dist" produced *two* disjoint tarballs? One of
which is guaranteed to be byte-for-byte identical to an archive of the
VCS at the release tag (in some clearly documented fashion; AIUI, "git
archive" does *not* do what we want).  The other contains all the files
that "autoreconf -i" or "./bootstrap.sh" or whatever would create, but
nothing else.  Diffs could be provided for both tarballs, or only for
the VCS-archive tarball, whichever turns out to be more compact (I can
imagine the diff for the generated-files tarball turning out to be
comparable in size to the generated-files tarball itself).


The way to do that is to detect that "make dist" is being run in a VCS 
checkout, ask the VCS which files are in version control, and assume the 
others were somehow "brought in" by autogen.sh or whatever.  The problem 
is that now Automake needs to start growing support for varying version 
control systems, unless we /really/ want to say that this feature only 
works with Git.


The problem is that now the disjoint tarballs both need to be unpacked 
in the same directory to build the package and once that is done, how 
does "make dist" rebuild the distribution it was run from?  The file 
lists would need to be stored in the generated-files tarball.


The other problem is that this really needs to be an option.  DejaGnu, 
for example, stores the Autotools-generated files in Git and releases 
are just snapshots of the working tree.  (DejaGnu can also now *run* 
from a Git checkout without actually installing it, but that is a 
convenience limited to interpreted languages.)


Lastly, publishing a modified (third-party) distribution derived from a 
release instead of VCS *is* permitted.  (I believe this is a case of 
freedom 3.)  How would this feature interact with that?



-- Jacob

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Russ Allbery wrote:

[...]

There is extensive ongoing discussion of this on debian-devel.  There's no
real consensus in that discussion, but I think one useful principle that's
emerged that doesn't disrupt the world *too* much is that the release
tarball should differ from the Git tag only in the form of added files.
  


From what I understand, the xz backdoor would have passed this check.  
The backdoor dropper was hidden in test data files that /were/ in the 
repository, and required code in the modified build-to-host.m4 to 
activate it.  The m4 files were not checked into the repository, instead 
being added (presumably by running autogen.sh with a rigged local m4 
file collection) while preparing the release.


Someone with a copy of a crocked release tarball should check if 
configure even had the backdoor "as released" or if the attacker was 
/depending/ on distributions to regenerate configure before packaging xz.



-- Jacob

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Zack Weinberg wrote:

[...] but I do think there's a valid point here: the malicious xz
maintainer *might* have been caught earlier if they had committed the
build-to-host.m4 modification to xz's VCS.


That would require someone to notice that xz.git has a build-to-host.m4 
that does not exist anywhere in the history of gnulib.git.  That is a 
fairly complex scan, although it does look straightforward to 
implement.  That said, the m4 files in Gnulib *are* Free Software, so 
having a modified version cannot itself raise too many concerns.



  (Or they might not have!
Witness the three (and counting) malicious patches that they barefacedly
submitted to *other* software and got accepted because the malice was
subtle enough to pass through code review.)
  


Exactly.  :-/

That said, the whole thing looks to me like the attackers were trying to 
/not/ hit the more (what is the best word?) "advanced" users---the 
backdoor would only be inserted if building distribution packages, and 
then only under dpkg or rpm, not other systems like Gentoo's Portage or 
in an unpackaged "./configure && make && sudo make install" build.  This 
would, of course, hit the most widely used systems, including (reports 
are that the sock farm tried very hard to get Ubuntu to ship the crocked 
version in their upcoming release, but the freeze was upheld) the 
systems most commonly used by less technically-skilled users, but 
pointedly exclude systems that require greater skill to use---and whose 
users would be more likely to notice anything amiss and start tearing 
the system apart with the debugger.  Unfortunately for Mr. Sockmaster, 
it turns out that some highly-skilled users *do* use the widely-used 
systems and the backdoor caused sshd to misbehave enough to draw 
suspicion.  (Profiling reports that sshd is spending most of its time in 
liblzma---a library it has no reason to use---will tend to raise a few 
eyebrows.  :-)  )



[...]
  
Maybe the best revision to the GNU Coding Standards would be that 
releases should, if at all possible, contain only text?  Any binary 
files needed for testing can be generated during "make check" if 
necessary



I don't think this is a good idea.  It's only a speed bump for someone
trying to smuggle malicious data into a package (think "base64 -d") and
it makes life substantially harder for honest authors of programs that
work with binary data, and authors of material whose "source code"
(as GPLv3 uses that term) *is* binary data.  Consider pngsuite, for
instance (http://www.schaik.com/pngsuite/) -- it would be a *ton* of
work to convert each of these test PNG files into GNU Poke scripts,
and probably the result would be *less* ergonomic for purposes of
improving the test suite.
  


That is a bad example because SNG (https://sng.sourceforge.net/>) 
exists precisely to provide a a text representation of PNG binary 
structures.  (Admittedly, if I recall correctly, the contents of IDAT 
are simply a hexdump.)


While we are on the topic, this leaves the other obvious place to hide 
binary data:  images used as part of the manual.  There is a reason that 
I added the "if at all possible" caveat, and I am not certain that it is 
always possible.



I would like to suggest that a more useful policy would be "files
written to $prefix by 'make install' should not have any data
dependency on files labeled as part of the package's testsuite".
That doesn't constrain honest authors and it seems within the
scope of what the reproducible builds people could test for.
(Build the package, install to nonce prefix 1, unpack the tarball
again, delete the test suite, build again, install to prefix 2, compare.)
Of course a sufficiently determined malicious coder could detect
the reproducible-build test environment, but unlike "no binary data"
this is a substantial difficulty increment.


This could be a good idea.  Another way to check this even without 
reproducible builds would be to ensure that the access timestamps on 
testsuite files do not change while "make" is processing the main 
sources.  Checking this is slightly more invasive, since you would need 
to run a hook between processing top-level directories during the main 
build, but for packages using recursive Automake, you could simply run 
"make -C src" (or wherever the main sources are) and make sure that the 
testsuite files still have the same atime afterwards.  I admit that this 
is harder to automate in general, but distribution packaging processes 
already have other metadata that is manually maintained, so identifying 
the source subtrees that yield the installable artifacts should not be 
difficult.


Now that I think about it, I suggest tightening that policy a bit 
further:  "files produced by make in the source subtree (typically src/) 
shall have no data dependency on files outside of that tree"


I doubt anyone ever thought that recursive make could end up as 
security/verifiability feature.  8-|



-- Jacob

Re: automated release building service

[Jacob Bachmeyer]

Bruno Haible wrote:

Jacob Bachmeyer wrote:
  

Essentially, this would be an automated release building service:  upon
request, make a Git checkout, run autogen.sh or equivalent, make dist,
and publish or hash the result.  The problem is that an attacker who
manages to gain commit access to a repository may be able to launch
attacks on the release building service, since "make dist" can run
scripts.  The service could probably mount the working filesystem noexec
since preparing source releases should not require running (non-system)
binaries and scripts can be run by directly feeding them into their
interpreters even if the filesystem is mounted noexec, but this still
leaves all available interpreters and system tools potentially available.



Well, it'd at least make things more difficult for the attacker, even
if it wouldn't stop them completely.
  
  
Actually, no, it would open a *new* target for attackers---the release 
building service itself.  Mounting the scratchpad noexec would help to 
complicate attacks on that service, but right now there is *no* central 
point for an attacker to hit to compromise releases.  If a central 
release building service were set up, it would be a target, and an 
attacker able to arrange a persistent compromise of the service could 
then tamper with later releases as they are built.  This should be 
fairly easy to catch, if an honest maintainer has a secure environment, 
("Why the  does the central release service tarball not match mine?  
And what the  is the extra code in this diff between its tarball 
and mine!?") but there is a risk that, especially for large projects, 
maintainers start relying on the central release service instead of 
building their own tarballs.


The problem here was not a maintainer with a compromised system---it 
seems that "Jia Tan" was a malefactor's sock puppet from the start.



There are several problems that such an automated release building service
would create. Here are a couple of them:

* First of all, it's a statement of mistrust towards the developer/maintainer,
  if developers get pressured into using an automated release building
  service rather than producing the tarballs on their own.
  This demotivates and turns off developers, and it does not fix the
  original problem: If a developer is in fact a malefactor, they can
  also commit malicious code; they don't need to own the release process
  in order to do evil things.
  


Limiting trust also limits the value of an attack as well, thus 
protecting the developers/maintainers from at least sane attackers in 
some ways.  I also think that this point misunderstands the original 
proposal (or I have misunderstood it).  To some extent, projects using 
Automake already have that automated release building service; we call 
it "make dist" and it is a distributed service running on each 
maintainer's machine, including distribution package maintainers who 
regenerate the Autotools files.  A compromise of a developer's machine 
is thus valuable as it allows to tamper with releases, but the risk is 
managed somewhat by each developer building only their own releases.


A central service as a "second opinion" would be a risk, but would also 
make those compromises even more difficult---now the attacker must hit 
both the central service *and* the dev box *and* coordinate to ensure 
that only packages prepared at the central service for which the 
maintainer's own machine is cracked are tampered, lest the whole thing 
be detected.  This is even harder on the attacker, which is a good 
thing, of course.


The more dangerous risk is that the central service becomes overly 
trusted and ceases to be merely the "second opinion" on a release.  If 
that occurs, not only would we be right back to no real check on the 
process, but now *all* the releases come from one place.  A compromise 
of the central release service would then allow *all* releases to be 
tampered, which is considerably more valuable to an attacker.



* Such an automated release building service is a piece of SaaSS. I can
  hardly imagine how we at GNU tell people "SaaSS is as bad as, or worse
  than, proprietary software" and at the same time advocate the use of
  such a service.
  


As long as it runs on published Free Software and anyone is free to set 
up their own instance, I would disagree here.  I think we need to work 
out where the line between "hosting" and "SaaSS" actually is, and I am 
not sure that it has a clear technical description, since SaaSS is 
ultimately an ethical issue.



* Like Jacob mentioned, such a service quickly becomes a target for
  attackers. So, instead of trusting a developer, you now need to trust
  the technical architecture and the maintainers of such a service.
  


I think I may know an example of something similar:  if I recall 
correctly, F-Droid original

Re: libsystemd dependencies

[Jacob Bachmeyer]

Bruno Haible wrote:

Jacob Bachmeyer wrote:
  
some of the blame for this needs to fall on the 
systemd maintainers and their "katamari" architecture.  There is no good 
reason for notifications of daemon startup to pull in liblzma, but using 
libsystemd for that purpose does exactly that, and ended up getting 
xz-utils targeted as a means of getting to sshd without the OpenSSH 
maintainers noticing.



The systemd people are working on reducing the libsystemd dependencies:
https://github.com/systemd/systemd/issues/32028

However, the question remains unanswered why it needs 3 different
compression libraries (liblzma, libzstd, and liblz4). Why would one
not suffice?
  


From reading other discussions, the only reason libsystemd pulls in 
compression libraries at all is its "katamari" architecture:  the 
systemd journal can be optionally compressed with any of those 
algorithms, and the support for reading the journal (which libsystemd 
also provides) therefore requires support for all of them.  No, sshd 
(even with the distribution patches at issue) does /not/ use that 
support whatsoever.


Better design would split libsystemd into separate libraries:  
libsystemd-notify, libsystemd-journal, etc.  I suspect that there are 
more logically distinct modules that have been "katamaried" into one 
excessively large library.  The C runtime library has an excuse for 
being such an agglomeration, but also note that libc has *zero* hard 
external dependencies.  You can ridicule NSS if you like, but NSS 
modules are only loaded if NSS is used.  (To be fair, sshd almost 
certainly /does/ use functions provided by NSS.)  The systemd developers 
do not have that excuse, and their library *has* external dependencies.


I believe the systemd developers cite convenience as justification for 
the practice, because apparently figuring out which libraries (out of a 
set partitioned based on functionality) you need to link is "too hard" 
for developers these days.  (Perhaps that is the real reason they want 
to replace X11?)  That "convenience" nearly got all servers on the 
Internet running the major distributions backdoored with critical 
severity and we do not yet know exactly what the backdoor blob did.  The 
preliminary reports that it was an RCE backdoor that would pass commands 
smuggled in public key material in SSH certificates to system(3) (as 
root of course, since that is sshd's context at that stage) are 
inconsistent with the slowdown that caused the backdoor to be 
discovered.  I doubt that SSH logins were using that code path, and the 
SSH scanning botnets almost certainly are not presenting certificates, 
yet it apparently (reports have been unclear on this point) was the 
botnet scanning traffic that led to the discovery of sshd wasting 
considerable CPU time in liblzma...


I am waiting for the proverbial other shoe to drop on that one.


-- Jacob

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Jose E. Marchesi wrote:

Jose E. Marchesi wrote:


[...]



I agree that distcheck is good but not a cure all.  Any static
system can be attacked when there is motive, and unit tests are
easily gamed.
  
  

The issue seems to be releases containing binary data for unit tests,
instead of source or scripts to generate that data.  In this case,
that binary data was used to smuggle in heavily obfuscated object
code.



As a side note, GNU poke (https://jemarch.net/poke) is good for
generating arbitrarily complex binary data from clear textual
descriptions.
  

While it is suitable for that use, at last check poke is itself very
complex, complete with its own JIT-capable VM.  This is good for
interactive use, but I get nervous about complexity in testsuites,
where simplicity can greatly aid debugging, and it /might/ be possible
to hide a backdoor similarly in a poke pickle.  (This seems to be a
general problem with powerful interactive editors.)



Yes, I agree simplicity it is very desirable, in testsuites and actually
everywhere else.  I also am not fond of dragging in dependencies.
  


Exactly---I am sure that poke is great for interactive use, but a 
self-contained solution is probably better for a testsuite.



But I suppose we also agree in that it is not possible to assembly
non-trivial binary data structures in a simple way, without somehow
moving the complexity of the encoding into some sort of generator, which
will not be simple.  The GDB testsuite, for example, ships with a DWARF
assembler written in around 3000 lines of Tcl.  Sure, it is simpler than
poke and doesn't drag in additional dependencies.  But it has to be
carefully maintained and kept up to date, and the complexity is there.
  


The problem for a compression tool testsuite is that compression formats 
are (I believe) defined as byte-streams or bit-streams.  Further, the 
generator(s) must be able to produce /incorrect/ output as well, in 
order to test error handling.



Further, GNU poke defines its own specialized programming language for
manipulating binary data.  Supplying generator programs in C (or C++)
for binary test data in a package that itself uses C (or C++) ensures
that every developer with the skills to improve or debug the package
can also understand the testcase generators.



Here we will have to disagree.

IMO it is precisely the many and tricky details on properly marshaling
binary data in general-purpose programming languages that would have
greater odds to lead to difficult to understand, difficult to maintain
and possibly buggy or malicious encoders.  The domain specific language
is here an advantage, not a liability.

This you need to do in C to encode and generate test data for a single
signed 32-bit NUMBER in an output file in a _more or less_ portable way:

  void generate_testdata (off_t offset, int endian, int number)
  {
int bin_flag = 0, fd;

  #ifdef _WIN32
int bin_flag = O_BINARY;
  #endif
fd = open ("testdata.bin", bin_flag, S_IWUSR);
if (fd == -1)
  fatal ("error generating data.");

if (endian == BIG)

  {
b[0] = (number >> 24) & 0xff;
b[1] = (number >> 16) & 0xff;
b[2] = (number >> 8) & 0xff;
b[3] = number & 0xff;
  }
else
  {
b[3] = (number >> 24) & 0xff;
b[2] = (number >> 16) & 0xff;
b[1] = (number >> 8) & 0xff;
b[0] = number & 0xff;
  }

lseek (fd, offset, SEEK_SET);
for (i = 0; i < 4; ++i)
  write (fd, [i], 1);
close (fd);
  }
  


While that is a nice general solution, (aside from neglecting the 
declaration "uint8_t b[4];"; with "int b[4];", the code would only work 
on a little-endian processor; with no declaration, the compiler will 
reject it) a compression format would be expected to define the 
endianess of stored values, so the major branch in that function would 
collapse to just one of its alternatives.  Compression formats are 
generally defined as streams, so a different decomposition of the 
problem would likely make more sense:  (example untested)


   void emit_int32le (FILE * out, int value)
   {
 unsigned int R, i;

 for (R = (unsigned int)value, i = 0; i < 4; R = R >> 8, i++)
   if (fputc(R & 0xff, out) == EOF)
 fatal("error writing int32le");
   }
 

Other code handles opening OUT, or OUT is actually stdout and we are 
writing down a pipe or the shell handled opening the file.  (The main 
function can easily check that stdout is not a terminal and bail out if 
it is.)  Remember that I am suggesting test generator programs, which do 
not need to be as general as ordinary code, nor do they need the same 
level of user-friendliness, since they are expected to be run from 
scripts that encode the precise knowledge of how to call them.  (That 
this version is also probably more efficient by avoiding a syscall for 
every byte written is irrelevant for its intended use.)



This is 

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Tomas Volf wrote:

On 2024-03-31 14:50:47 -0400, Eric Gallager wrote:
  

With a reproducible build system, multiple maintainers can "make dist"
and compare the output to cross-check for erroneous / malicious dist
environments.  Multiple signatures should be harder to compromise,
assuming each is independent and generally trustworthy.


This can only work if a package /has/ multiple active maintainers.
  

Well, other people besides the maintainers can also run `make dist`
and `make distcheck`. My idea was to get end-users in the habit of
running `make distcheck` themselves before installing stuff. And if
that's too much to ask of end users, I'd also point out that there are
multiple kinds of maintainer: besides the upstream maintainer, there
are also usually separate distro maintainers. Even if there's only 1
upstream maintainer, as was the case here, I still think that it would
be good to get distro maintainers in the habit of including `make
distcheck` as part of their own release process, before they accept
updates from upstream.



What would be helpful is if `make dist' would guarantee to produce the same
tarball (bit-to-bit) each time it is run, assuming the tooling is the same
version.  Currently I believe that is not the case (at least due to timestamps).
  


A "tardiff" tool that ignores timestamps would be a solution to that 
problem, but not to this backdoor.



Combined with GNU Guix that would allow simple way to verify that `make dist'
was used, and the resulting artifact not tampered with, even without any central
signing.


The Guix "challenge" operation would not have detected this backdoor 
because *it* *was* *in* *the* *upstream* *release*.  The build service 
works from that release tarball and you build from that same release 
tarball.  GNU Guix ensures an equivalent build environment and your 
results *will* match---either the backdoor was not inserted or it was 
inserted in both builds.



The flow of the attack as I understand it was:

   (0)  (speculation on motivation) The attacker wanted a "Golden Key" 
to SSH and started looking for ways to backdoor sshd.
   (1)  The attacker starts a sockpuppet campaign and manages to get 
one of his sockpuppets appointed co-maintainer of xz-utils.
   (2)  [2023-06-27] The sockpuppet merges a pull request believed to 
be from another sockpuppet in commit 
ee44863ae88e377a5df10db007ba9bfadde3d314.
   (3)  [2024-02-15] The sockpuppet "updates m4/.gitignore" to add 
build-to-host.m4 to the list in commit 
4323bc3e0c1e1d2037d5e670a3bf6633e8a3031e.
   (4)  [2024-02-23] The sockpuppet adds 5 files to the xz-utils 
testsuite in commit cf44e4b7f5dfdbf8c78aef377c10f71e274f63c0.
   (5)  [2024-03-08] To cover tracks, the sockpuppet finally adds a 
test using bad-3-corrupt_lzma2.xz in commit 
a3a29bbd5d86183fc7eae8f0182dace374e778d8.
   (6)  [2024-03-08] The sockpuppet revises two of those files with a 
lame excuse in commit a3a29bbd5d86183fc7eae8f0182dace374e778d8.


The quick analysis of the Git history supporting steps 2 - 6 above has 
turned up another interesting detail:  no version of configure.ac 
actually committed ever used the gl_BUILD_TO_HOST macro.  An analysis 
found on pastebin noted that build-to-host.m4 is a dependency of 
gettext.m4.  Following up finds commit 
3adaddd73c8edcceaed059e859bd5262df65fc5a of 2023-02-18 in the GNU 
gettext repository introduced the use of gl_BUILD_TO_HOST, apparently as 
part of moving some existing path translation logic to gnulib and 
generalizing it for use elsewhere.  This commit is innocent (it is 
*extremely* unlikely that Bruno Haible was involved in the backdoor 
campaign) and also explains why the backdoor was checking for "dnl 
Convert it to C string syntax." in m4/gettext.m4:  that comment was 
removed in the same commit that switch to using gl_BUILD_TO_HOST.  The 
change to gettext also occurred about a year before the sockpuppet began 
to take advantage of it.


It almost "feels like" the attacker was waiting for an opportunity to 
make plausible changes to autoconf macros and finally got one when 
updating the m4/ files for the 5.6.0 release.  Could someone with the 
release tarballs confirm that m4/gettext.m4 was updated between 
v5.5.2beta and v5.6.0?  I doubt the entire backdoor was developed in the 
week between those two commits.  In fact, the timing around introducing 
ifuncs suggests to me that the binary blob was at least well into 
development by mid-2023.


The commit message at step 2 claims that using ifuncs with 
-fsanitize=address causes segfaults.  If this is true generally, the 
glibc team should probably reconsider whether the abuse potential is 
worth the benefit of the feature and possibly investigate how the 
feature was introduced to glibc.  If this was an excuse, it provided a 
clever way to prevent oss-fuzz from finding the backdoor, as disabling 
ifuncs provides a conveniently hidden flag to disable the backdoor.


While double-checking the above, I 

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Eric Gallager wrote:

On Sun, Mar 31, 2024 at 3:20 AM Jacob Bachmeyer  wrote:
  

dherr...@tentpost.com wrote:


[...]

The issue seems to be releases containing binary data for unit tests,
instead of source or scripts to generate that data.  In this case, that
binary data was used to smuggle in heavily obfuscated object code.

[...]



Maybe this is something that the GNU project could start making
stronger recommendations about.
  


The key issue seems to be generating binary test data during `make` or 
`make check`, using GNU poke, GNU Awk, Perl, Tcl, small C programs, or 
something else, instead of packaging it in the release.  The xz-utils 
backdoor was smuggled into the repository wrapped in compressed test data.



With a reproducible build system, multiple maintainers can "make dist"
and compare the output to cross-check for erroneous / malicious dist
environments.  Multiple signatures should be harder to compromise,
assuming each is independent and generally trustworthy.
  

This can only work if a package /has/ multiple active maintainers.



Well, other people besides the maintainers can also run `make dist`
and `make distcheck`. My idea was to get end-users in the habit of
running `make distcheck` themselves before installing stuff. And if
that's too much to ask of end users, I'd also point out that there are
multiple kinds of maintainer: besides the upstream maintainer, there
are also usually separate distro maintainers. Even if there's only 1
upstream maintainer, as was the case here, I still think that it would
be good to get distro maintainers in the habit of including `make
distcheck` as part of their own release process, before they accept
updates from upstream.
  


The problem with that is that `make distcheck` only verifies that the 
working tree can produce a reasonable release tarball.  The backdoored 
xz-utils releases *would* *have* *passed* *this* *test* as far as I can 
determine.  It catches errors like omitting files from the lists in 
Makefile.am.  It will *not* catch a modified m4 file or questionable 
test data that has been properly listed as part of the release.



Maybe GNU should establish a cross-verification signing standard and
"dist verification service" that automates this process?  Point it to
a repo and tag, request a signed hash of the dist package...  Then
downstream projects could check package signatures from both the
maintainer and such third-party verifiers to check that nothing was
inserted outside of version control.
  

Essentially, this would be an automated release building service:  upon
request, make a Git checkout, run autogen.sh or equivalent, make dist,
and publish or hash the result.  The problem is that an attacker who
manages to gain commit access to a repository may be able to launch
attacks on the release building service, since "make dist" can run
scripts.  The service could probably mount the working filesystem noexec
since preparing source releases should not require running (non-system)
binaries and scripts can be run by directly feeding them into their
interpreters even if the filesystem is mounted noexec, but this still
leaves all available interpreters and system tools potentially available.



Well, it'd at least make things more difficult for the attacker, even
if it wouldn't stop them completely.
  


Actually, no, it would open a *new* target for attackers---the release 
building service itself.  Mounting the scratchpad noexec would help to 
complicate attacks on that service, but right now there is *no* central 
point for an attacker to hit to compromise releases.  If a central 
release building service were set up, it would be a target, and an 
attacker able to arrange a persistent compromise of the service could 
then tamper with later releases as they are built.  This should be 
fairly easy to catch, if an honest maintainer has a secure environment, 
("Why the  does the central release service tarball not match mine?  
And what the  is the extra code in this diff between its tarball 
and mine!?") but there is a risk that, especially for large projects, 
maintainers start relying on the central release service instead of 
building their own tarballs.


The problem here was not a maintainer with a compromised system---it 
seems that "Jia Tan" was a malefactor's sock puppet from the start.



-- Jacob

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Jose E. Marchesi wrote:

[...]


I agree that distcheck is good but not a cure all.  Any static
system can be attacked when there is motive, and unit tests are
easily gamed.
  

The issue seems to be releases containing binary data for unit tests,
instead of source or scripts to generate that data.  In this case,
that binary data was used to smuggle in heavily obfuscated object
code.



As a side note, GNU poke (https://jemarch.net/poke) is good for
generating arbitrarily complex binary data from clear textual
descriptions.


While it is suitable for that use, at last check poke is itself very 
complex, complete with its own JIT-capable VM.  This is good for 
interactive use, but I get nervous about complexity in testsuites, where 
simplicity can greatly aid debugging, and it /might/ be possible to hide 
a backdoor similarly in a poke pickle.  (This seems to be a general 
problem with powerful interactive editors.)


Further, GNU poke defines its own specialized programming language for 
manipulating binary data.  Supplying generator programs in C (or C++) 
for binary test data in a package that itself uses C (or C++) ensures 
that every developer with the skills to improve or debug the package can 
also understand the testcase generators.



-- Jacob

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

dherr...@tentpost.com wrote:

On 2024-03-30 18:25, Bruno Haible wrote:

Eric Gallager wrote:


Hm, so should automake's `distcheck` target be updated to perform
these checks as well, then?


The first mentioned check can not be automated. ...

The second mentioned check could be done by the maintainer, ...



I agree that distcheck is good but not a cure all.  Any static system 
can be attacked when there is motive, and unit tests are easily gamed.


The issue seems to be releases containing binary data for unit tests, 
instead of source or scripts to generate that data.  In this case, that 
binary data was used to smuggle in heavily obfuscated object code.


The best analysis in one place that I have found so far is 
https://gynvael.coldwind.pl/?lang=en=782>.  In brief, grep is 
used to locate the main backdoor files by searching for marker strings.  
After running tests/files/bad-3-corrupt_lzma2.xz through tr(1), it 
becomes a /valid/ xz file that decompresses to a shell script that 
extracts a second shell script from part of the compressed data in 
tests/files/good-large_compressed.lzma and pipes it to a shell.  That 
second script has two major functions:  first, it searches the test 
files for four six-byte markers, and it then extracts and decrypts 
(using a simple RC4-alike implemented in Awk) the binary backdoor also 
found in tests/files/good-large_compressed.lzma.  The six-byte markers 
mark beginning and end of raw LZMA2 streams obfuscated with a simple 
substitution cipher.  Any such streams found would be decompressed and 
read by the shell, but neither of the known crocked releases had any 
files containing those markers.  The binary backdoor is an x86-64 object 
that gets unpacked into liblzma_la-crc64-fast.o, unless m4/gettext.m4 
contains "dnl Convert it to C string syntax." which is a clever flag 
because about no one actually checks that those m4 files in release 
tarballs actually match what the GNU project distributes.  The object 
itself is just the backdoor and presumably provides the symbol 
_get_cpuid as its entrypoint, since the unpacker script patches the 
src/liblzma/check/crc{64,32}_fast.c files in a pipeline to add calls to 
that function and drops the compiled objects in .libs/.  Running make 
will then skip building those objects, since they are already 
up-to-date, and the backdoored objects get linked into the final binary.


Commit 6e636819e8f070330d835fce46289a3ff72a7b89 
(https://git.tukaani.org/?p=xz.git;a=commitdiff;h=6e636819e8f070330d835fce46289a3ff72a7b89>) 
was an update to the backdoor.  The commit message is suspicious, 
claiming the use of "a constant seed" to generate reproducible test 
files, but /not/ declaring how the files were produced, which of course 
prevents reproducibility.


With a reproducible build system, multiple maintainers can "make dist" 
and compare the output to cross-check for erroneous / malicious dist 
environments.  Multiple signatures should be harder to compromise, 
assuming each is independent and generally trustworthy.


This can only work if a package /has/ multiple active maintainers.

You also have a small misunderstanding here:  "make dist" prepares a 
(source) release tarball, not a binary build, so this is a 
closely-related issue but actually distinct from reproducible builds.  
Also easier to solve, since we only have to make the source tarball 
reproducible.


Maybe GNU should establish a cross-verification signing standard and 
"dist verification service" that automates this process?  Point it to 
a repo and tag, request a signed hash of the dist package...  Then 
downstream projects could check package signatures from both the 
maintainer and such third-party verifiers to check that nothing was 
inserted outside of version control.


Essentially, this would be an automated release building service:  upon 
request, make a Git checkout, run autogen.sh or equivalent, make dist, 
and publish or hash the result.  The problem is that an attacker who 
manages to gain commit access to a repository may be able to launch 
attacks on the release building service, since "make dist" can run 
scripts.  The service could probably mount the working filesystem noexec 
since preparing source releases should not require running (non-system) 
binaries and scripts can be run by directly feeding them into their 
interpreters even if the filesystem is mounted noexec, but this still 
leaves all available interpreters and system tools potentially available.



-- Jacob

Re: GNU Coding Standards, automake, and the recent xz-utils backdoor

[Jacob Bachmeyer]

Eric Gallager wrote:


Specifically, what caught my attention was how the release tarball
containing the backdoor didn't match the history of the project in its
git repository. That made me think about automake's `distcheck`
target, whose entire purpose is to make it easier to verify that a
distribution tarball can be rebuilt from itself and contains all the
things it ought to contain.


The problem is that a release tarball is a freestanding object, with no 
dependency on the repository from which it was produced.  In this case, 
the attacker added a bogus "update" of build-to-host.m4 from gnulib to 
the release tarball, but that file is not stored in the Git repository.  
This would not have tripped "make distcheck" because the crocked tarball 
can indeed be used to rebuild another crocked tarball.


As Alexandre Oliva mentioned in his reply, there is not really any good 
way to prevent this, since the attacker could also patch the generated 
configure script more directly.  (I seem to remember past incidents 
where tampered release tarballs had configure scripts that would 
download and run shell scripts.  If you ran configure as root, well...)  
The *user* could catch issues like this backdoor, since the backdoor 
appears (based on what I have read so far) to materialize certain object 
files while configure is running, while `find . -iname '*.o'` /should/ 
return nothing before make is run.  This also suggests that running 
"make clean" after configure would kill at least this backdoor.  A 
*very* observant (unreasonably so) user might notice that "make" did not 
build the objects that the backdoor provided.


Of course, an attacker could sneak around this as well by moving the 
process for unpacking the backdoor object to a Makefile rule, but that 
is more likely to "stick out" to an observant user, as well as being an 
easy target for automated analysis ("Which files have 'special' rules?") 
since you cannot obfuscate those from make(1) and expect them to still 
work.  In this case, the backdoor was ultimately discovered when it 
caused performance problems in sshd, which should not be using liblzma 
at all, but gets linked with it courtesy of libsystemd on major 
GNU/Linux distributions.  Yes, this means that systemd is a contributing 
factor to this incident, and that is aggravated by its unnecessary use 
of excessive dependencies.  (Sending a notification that a daemon is 
ready should /not/ require compression support of any type.  The 
"katamari" architecture model used in systemd had the effect here of 
broadening the supply-chain attack surface for OpenSSH sshd to include 
xz-utils, which is insane.)


The bulk of the attack payload seems to have been stored in the Git 
repository, disguised as binary test data in files 
tests/files/{bad-3-corrupt_lzma2.xz,good-large_compressed.lzma}.  The 
modified build-to-host.m4 merely added code to configure to start the 
process of unpacking the backdoor.  In a build from Git, the legitimate 
build-to-host.m4 would get copied in from gnulib and the backdoor would 
remain hidden.


Maybe the best revision to the GNU Coding Standards would be that 
releases should, if at all possible, contain only text?  Any binary 
files needed for testing can be generated during "make check" if 
necessary, with generator programs packaged (as source or scripts) in 
the release.



-- Jacob

[bug#68416] [PATCH] python: add 3.20 - 3.16 to the version search list

[Jacob Bachmeyer]

Frederic Berat wrote:

On Sun, Jan 14, 2024 at 3:13 AM Karl Berry  wrote:

  

With Python 3.12 out now, and 3.13 out in ~9 months, the existing
runway
is running out.  Bump up to 3.20 for the next Automake release.

Applied, thanks.

Not proposing to try anything for our upcoming release, but I wonder if
there is some more general way to handle Python versions? We don't have
to laboriously list every possible version for anything else.




That may not be the most clever way to do it, but you can probably build
the list dynamically at least, with something like (untested):

pythons="python python2 python3"
for i in {20..0};do pythons="$pythons python3.$i";done
for i in {7..0};do pythons="$pythons python2.$i";done

m4_define_default([_AM_PYTHON_INTERPRETER_LIST], [$pythons])
  


The {20..0} is a syntax unfamiliar to me, yet even Bash 3 has it... 
interesting.  Also interesting that it is brace expansion, so you could do:


pythons=`echo python{,2,3} python3.{20..0} python2.{7..0}`

The use of command substitution and echo works around the small catch 
that brace expansion does not occur inside double quotes.  However, 
brace expansion does occur inside a command substitution inside double 
quotes, but the result of command substitution as a variable assignment 
is not subject to word splitting, so the double quotes can be omitted.  
I wonder if all shells that allow {20..0} also allow $() for command 
substitution, or if this code must be valid in shells that predate POSIX...


Oh wait, after reviewing the thread, I find that this is in m4 macros.  
Yes, it could be dynamically produced, but not using shell iteration 
constructs or expansions.  Macros in m4 are allowed to use tail 
recursion, and counted iteration is possible to implement (and autoconf 
provides some macros for it).


The required quoting is "fun" and Automake may complain that this 
appears to be underquoted:  (untested; the fine details of the quoting 
may be wrong)


m4_define_default([_AM_PYTHON_INTERPRETER_LIST],  dnl
   [python python2 python3] dnl
   m4_for(i, 20, 0, -1, [ python3.i]) dnl
   m4_for(i, 7, 0, -1, [ python2.i]))

As written, this might lead to some extra spaces in the interpreter 
list, but the shell will ignore those when running configure.



-- Jacob

[bug#67841] [PATCH] Clarify error messages for misuse of m4_warn and --help for -W.

[Jacob Bachmeyer]

Karl Berry wrote:
"All possible characters have a UTF-8 representation so this function 
[encode_utf8] cannot fail."


What about non-characters, i.e., byte sequences that are invalid UTF-8?
  


Each individual byte gets encoded as UTF-8.  0x00..0x7F are an identity 
map, while 0x80..0xFF are translated to 2-octet sequences.  /Decoding/ 
UTF-8 can blow up or produce bogus results (I think Perl might just drop 
in the "substitute" character and emit a warning) but /encoding/ UTF-8 
always works, even on UTF-8.  Remember that Perl could handle arbitrary 
binary data long before it had Unicode support.



What I found was that using \N{...} implies a Unicode string. From the
charnames(3) man page (stranged not named "perlcharnames"):

 Otherwise, any string that includes a "\N{charname}" or "\N{U+code
 point}" will automatically have Unicode rules (see "Byte and
 Character Semantics" in perlunicode).
  


That page is named "charnames" because it documents the "charnames" 
pragmatic module.  The man page version was translated from the perldoc 
system when perl was built/installed/packaged.  The "perlunicode" page 
documents general Unicode support in Perl.



Maybe pack("C") somehow can get to the bytes from a Unicode string?
  


All strings in Perl are Unicode now, internally stored as UTF-8 or, as 
an optimization if no codepoints exceed 255, raw octets.  (A string of 
raw octets is considered to be a sequence of characters in the range 
[0,255].)  The "utf8 flag" on a string indicates which of those forms is 
in use on any particular string.  Using encode_utf8 simply gives you the 
internal encoding, converting an octet string to UTF-8 if needed, marked 
as an octet string.  If the string is already UTF-8, encode_utf8 simply 
clears the utf8 flag so you get access to the raw bytes.  (Brain twisted 
yet?  Mine was when I first looked at this...)


Perl's Unicode handling is fun because Perl could always handle binary 
data, and Unicode support was more-or-less retrofitted on top of that 
support for binary data.  In other words, if your program does not 
handle Unicode properly (or if you are running on Perl 5.6 and your 
program does not do the Perl 5.6 magic Unicode dances), Perl will treat 
"Unicode" data as its underlying octet sequence; thus my earlier advice 
to conditionally import Encode and shim encode_utf8 with an identity 
function if Encode is not available.



-- Jacob

[bug#67841] [PATCH] Clarify error messages for misuse of m4_warn and --help for -W.

[Jacob Bachmeyer]

Zack Weinberg wrote:

On Fri, Dec 15, 2023, at 7:08 PM, Jacob Bachmeyer wrote:
  

Zack Weinberg wrote:


[...]
Also, there’s a perl 2.14ism in one place (s///a) which I need
to figure out how to make 2.6-compatible before it can land.
  

...
  

+  $q_channel =~ s/([^\x20-\x7e])/"\\x".sprintf("%02x", ord($1))/aeg;
  

...
  
If I am reading perlre correctly, you should be able to simply drop the 
/a modifier because it has no effect on the pattern you have written, 
since you are using an explicit character class and are *not* using the 
/i modifier.



Thanks, you've made me realize that /a wasn't even what I wanted in the
first place.  What I thought /a would do is force s/// to act byte by
byte -- or, in the terms of perlunitut, force the target string to be
treated as a binary string.  That might be clearer with a concrete example:

$ perl -e '$_ = "\xE2\x88\x85"; s/([^\x20-\x7e])/sprintf("\\x%02x", ord($1))/eg; print 
"$_\n";'
\xe2\x88\x85
$ perl -e '$_ = "\N{EMPTY SET}"; s/([^\x20-\x7e])/sprintf("\\x%02x", ord($1))/eg; print 
"$_\n";'
\x2205

What change do I need to make to the second one-liner to make it also
print \xe2\x88\x85?


Add -MEncode to the one-liner and insert "$_ = encode_utf8($_);" before 
the substitution to declare that you want the string as UTF-8 bytes.  
The Encode documentation states:
"All possible characters have a UTF-8 representation so this function 
[encode_utf8] cannot fail."


In the actual patch, try "my $q_channel = encode_utf8($channel);" when 
initially copying the channel name.



  How do I express that in a way that is backward
compatible all the way to 5.6.0?


Now the fun part... Perl 5.6 had serious deficiencies in Unicode 
support; Encode was introduced with 5.8.  You will need to make the 
Encode import conditional and generate a stub for encode_utf8 if the 
import fails.  This should not be a problem since non-ASCII here in the 
first place is unlikely, and I think Perl 5.6 would treat non-ASCII as 
exactly the octet string you want anyway.


Something like:  (untested)

BEGIN {
 my $have_Encode = 0;
 eval { require Encode; $have_Encode = 1; };
 if ($have_Encode) {
   Encode->import('encode_utf8');
 } else {
   # for Perl 5.6, which did not really have Unicode support anyway
   eval 'sub encode_utf8 { return pop }';
 }
}

Note that the stub is defined using eval STRING rather than eval BLOCK 
because "sub" has compile-time effects in Perl and we only want it if 
Encode could not be loaded.



  And finally, how do I ensure that there is absolutely nothing I can put in 
the initial assignment to $_ that will cause the rest of the one-liner to 
crash?  For example
over in the Python universe it's very easy to get Unicode conversion
to crash:

$ python3 -c 'print("\uDC00".encode("utf-8"))'
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'utf-8' codec can't encode character '\udc00' in position 
0: surrogates not allowed
  


Not a problem in Perl:

$ perl -MEncode -e '$_ = "\x{dc00}"; $_ = encode_utf8($_); 
s/([^\x20-\x7e])/sprintf("\\x%02x", ord($1))/eg; print "$_\n";'

\xed\xb0\x80

:-)

-- Jacob

[bug#67841] [PATCH] Clarify error messages for misuse of m4_warn and --help for -W.

[Jacob Bachmeyer]

Zack Weinberg wrote:

[...]

Since this touches code shared between Autoconf and Automake, I’m not
checking it in yet and I’m requesting comments from both sides of the
fence.  Also, there’s a perl 2.14ism in one place (s///a) which I need
to figure out how to make 2.6-compatible before it can land.

[...]
 
+sub report_bad_channel ($$)

+{
+  my ($channel, $location) = @_;
+  my $message;
+  my $report_as = 'error';
+
+  # quotemeta is both too aggressive (e.g. it escapes '-') and
+  # too generous (it turns control characters into \ + themselves,
+  # not into symbolic escapes).
+  my $q_channel = $channel;
+  $q_channel =~ s/\\//g;
+  $q_channel =~ s/([^\x20-\x7e])/"\\x".sprintf("%02x", ord($1))/aeg;
+  $q_channel =~ s/(?=[\"\$\'\@\`])/\\/g;
+  $q_channel = '"' . $q_channel . '"';

[...]
  


If I am reading perlre correctly, you should be able to simply drop the 
/a modifier because it has no effect on the pattern you have written, 
since you are using an explicit character class and are *not* using the 
/i modifier.  (The documentation says that /a only affects the \d, \s, 
\w Perl character classes and the POSIX character classes.)  Further, 
Perl's sprintf is indeed sprintf, so you should be able to rewrite that 
substitution as:


s/([^\x20-\x7e])/sprintf('\\x%02x', ord $1)/eg


This slightly simplifies the program (eliminating one concatenation), 
which is probably of trivial consequence, since this code is on an error 
path and not an inner loop.  More importantly, including the "\\x" in 
the sprintf format string makes the intended output more clear.  The 
single quotes are both a (very slight) performance optimization and 
serve to document that no direct interpolation is intended because this 
is a format string.  Perl's handling of backslash in single quotes is 
fun:  backslash escapes itself but will also be taken literally if it 
does not form an escape that is allowed in q[].



-- Jacob

Re: [RFC PATCH]: autom4te: report subsecond timestamp support in --version

[Jacob Bachmeyer]

Zack Weinberg wrote:

On Mon, Dec 4, 2023, at 7:26 PM, Jacob Bachmeyer wrote:
  

Now that I have seen the actual patch, yes, this test should be
accurate.  The test in the main autom4te script will also work, even
if there is a mismatch between the script and its library



Good.

  

This appears to be misaligned with the GNU Coding Standards, which
states:  "The first line is meant to be easy for a program to parse;
the version number proper starts after the last space."

Perhaps the best option would be to conditionally add a line "This
autom4te supports subsecond timestamps." after the license notice?



I don't like putting anything after the license notice because it's
convenient to be able to pipe --version output to sed '/Copyright/,$d'
without losing anything relevant for troubleshooting.  So how about

$ autom4te --version
autom4te (GNU Autoconf) 2.71
Features: subsecond-timestamps

Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+/Autoconf: GNU GPL version 3 or later
<https://gnu.org/licenses/gpl.html>, <https://gnu.org/licenses/exceptions.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Akim Demaille.

This preserves the effectiveness of sed '/Copyright/,$d' and also
leaves room for future additions to the "Features:" line.


That looks like a good idea to me, although the GNU Coding Standards do 
say (section 4.8.1, "--version") that the copyright and license notices 
"should" immediately follow the version numbers.  The presence or 
absence of this feature is effectively determined by something similar 
to a library version (the availability of the Perl Time::HiRes module) 
and it is expected to be important for debugging, which is the criteria 
stated for listing library versions.  Further, "should" does not express 
an absolute requirement and no rationale that would effectively make an 
absolute requirement (like a rule for automated parsing) is given here, 
unlike for the version in the first line.



-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Zack Weinberg wrote:

On Mon, Dec 4, 2023, at 7:14 PM, Jacob Bachmeyer wrote:
  

Zack Weinberg wrote:


[snip everything addressed in the other thread]
  

Yes, there was a bit of confusion here; not only is the FileUtils
module synchronized between autom4te and automake



Thanks for reminding me that I need to make sure all those files are
actually in sync before I cut the final 2.72 release.

  

  require Time::HiRes;
  import Time::HiRes qw(stat);
  

I believe that the import is not actually necessary



The previous line is a "require", not a "use", so I believe it _is_
necessary.  Have I misunderstood?

  

... should do no harm as long as any use of stat in the code
is prepared to handle floating-point timestamps.



There's only one use, in 'sub mtime', and that's the place
where we actively want the floating-point timestamps.


Yes, before seeing your actual patch, I had the mistaken impression that 
this code was in autom4te itself, not the FileUtils module.  The import 
is needed in the FileUtils module, so the patch is correct.



-- Jacob

Re: [RFC PATCH]: autom4te: report subsecond timestamp support in --version

[Jacob Bachmeyer]

Zack Weinberg wrote:

The Automake test suite wants this in order to know if it’s safe to
reduce the length of various delays for the purpose of ensuring files
in autom4te.cache are newer than the corresponding source files.

* lib/Autom4te/FileUtils.pm: Provide (but do not export) a flag
  $subsecond_mtime, indicating whether the ‘mtime’ function reports
  modification time with precision greater than one second.
  Reorganize commentary and import logic for clarity.  Add
  configuration for emacs’ perl-mode to the bottom of the file.
  


Now that I have seen the actual patch, yes, this test should be 
accurate.  The test in the main autom4te script will also work, even if 
there is a mismatch between the script and its library, since Perl 
accepts a fully-qualified variable name even if that variable has never 
been declared; its value is undef, which is falsish in Boolean context.



* bin/autom4te.in ($version): If $Autom4te::FileUtils::subsecond_mtime
  is true, add the text “ (subsecond timestamps supported)” to the
  first line of --version output.
  


This appears to be misaligned with the GNU Coding Standards, which 
states:  "The first line is meant to be easy for a program to parse; the 
version number proper starts after the last space."


Perhaps the best option would be to conditionally add a line "This 
autom4te supports subsecond timestamps." after the license notice?



-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Zack Weinberg wrote:

On Sun, Dec 3, 2023, at 4:49 PM, Karl Berry wrote:
  
There would not need to be much parsing, just "automake --version | grep 
  
> HiRes" in that case, or "case `automake --version` in *HiRes*) ...;; 
> easc" to avoid running grep if you want.


I specifically want to hear what Karl thinks.

I lean towards Jacob's view that automake --version | grep HiRes will
suffice. Not having a new option seems simpler/better in terms of later
understanding, too. --thanks, karl.



Did I misunderstand which program's --version output we are talking about?
I thought we were talking about automake's testsuite probing the behavior
of *autom4te*, but all the quoted text seems to be imagining a probe of
*automake* instead.


Yes, there was a bit of confusion here; not only is the FileUtils module 
synchronized between autom4te and automake, but those two are near 
"sound-alikes" as I read them.  Oops.


The issue here seems to be determining if a fix that (I think) 
originated in automake has been applied to the active autom4te.



[...]

I'm not using the identifier "HiRes" because the use of Time::HiRes is an
implementation detail that could change.  For instance, if there's a third
party CPAN module that lets us get at nanosecond-resolution timestamps
*without* loss of precision due to conversion to an NV (aka IEEE double)
we could, in the future, look for that first.
  


That is fine, but "[HiRes]" or "[HiResTime]" is much shorter and we 
could use it as the name of the feature regardless of the underlying 
implementation.  Characters in the first line of `autom4te --version` 
are a finite resource if we want it to fit on a standard 80-column 
terminal without wrapping.  If we need to distinguish, "[HiRes] 
[HiRes-ns]" could be used to indicate your hypothetical integer 
nanosecond-resolution timestamp support, which would indicate also 
having sub-second timestamp support.


I also suggest changing the tag, since the GNU Coding Standards call for 
the version number to be indicated by the last space, but parenthesized 
text between the name and version is supposed to be the package, so this 
would lead to:


$ ./tests/autom4te --version
autom4te [HiResTime] (GNU Autoconf) 2.72d.6-49ab3-dirty
Copyright (C) 2023 Free Software Foundation, Inc.
[...]


Is this workable all the way around, everyone?  Or should the feature be 
indicated with another line after the license notice?  ("This autom4te 
has subsecond timestamp resolution.")  My apologies for neglecting to 
check this before suggesting a tag in the --version output.



The implementation is just

BEGIN
{
  our $subsecond_timestamps = 0;
  eval
{
  require Time::HiRes;
  import Time::HiRes qw(stat);
  $subsecond_timestamps = 1;
}
}

Jacob, can you confirm that's an accurate test, given all the things you
said earlier about ways that grepping the source code might get it wrong?


That will determine if (a) Time::HiRes could be loaded and (b) 
Time::HiRes::stat could be imported.  This is the same test that 
Autom{ak,4t}e::FileUtils effectively uses to use Time::HiRes::stat.  I 
believe that the import is not actually necessary (i.e. Time::HiRes 
always exported Time::HiRes::stat) but it should do no harm as long as 
any use of stat in the code is prepared to handle floating-point 
timestamps.  As long as the autom4te script and its library are 
consistent (which is the distribution's problem if they screw that up), 
this test should be accurate.



-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Karl Berry wrote:
> There would not need to be much parsing, just "automake --version | grep 
> HiRes" in that case, or "case `automake --version` in *HiRes*) ...;; 
> easc" to avoid running grep if you want.


I specifically want to hear what Karl thinks.

I lean towards Jacob's view that automake --version | grep HiRes will
suffice. Not having a new option seems simpler/better in terms of later
understanding, too. --thanks, karl.

P.S. As for case vs. grep, personally I find a simple if...grep easier
to comprehend/test/debug than a case statement. (Especially the
macro-ized AS_CASE, which just makes me have to look up its syntax every
time I see it.) Also fewer lines of source. Granted calling the external
grep is less efficient, but that seems insignificant to me. I understand
Paul and others may disagree ...


I agree that if...grep is more direct.  I suggested the case alternative 
because it stands out in my memory after I needed it once, but I do not 
recall exactly why that contortion was needed.


In configure, the efficiency difference is trivial because configure 
already runs many, many, many subprocesses.  One more grep will not make 
a difference on any reasonable platform.



-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Mike Frysinger wrote:

On 02 Dec 2023 17:07, Jacob Bachmeyer wrote:
  

Mike Frysinger wrote:


On 06 Apr 2023 21:29, Jacob Bachmeyer wrote:
  

Karl Berry wrote:


jb> a more thorough test would locate the autom4te script and grep it
for the perllibdir that was substituted when autoconf was
configured.

I guess that would work.
  
  

Challenge accepted.  Here's a refined version:  (lines \-folded for email)

if $PERL -I${autom4te_perllibdir:-$(sed -n \
  '/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}' \
<$(command -v autom4te))} -MAutom4te::FileUtils \
 -e 'exit defined $INC{q[Time/HiRes.pm]} ? 0 : 1'; then
   # autom4te uses Time::HiRes
else
   # autom4te does not use Time::HiRes
fi



this doesn't work on systems that wrap `autom4te`.  [...]

[...]

so i don't know why we would need to set/export autom4te_perllibdir in our
wrapper.  we've been doing this for over 20 years without ever setting that
var (or any other internal autoconf/automake var), so i'm pretty confident
our approach is OK.
  


No, not in the wrapper---in the automake ebuild script that runs 
configure to match the autom4te that the wrapper will run.  That test I 
proposed checks for autom4te_perllibdir in the environment before 
extracting it from autom4te precisely so distributions like Gentoo would 
have a knob to turn if their packaging breaks that test.


That said, it turns out that this whole line of discussion is now a red 
herring; see below.



[...]
i'm not aware of anything loading the Autom4te perl modules outside of the
autoconf projects.  does that actually happen ?  i don't think we want to
have automake start loading autoconf perl modules directly.  going through
the CLI interface seems better at this point.


Autoconf and Automake are very closely associated; there is even some 
shared code that is synchronized between them.  Autom4te::FileUtils is 
also Automake::FileUtils, for example.


The test we are discussing here was intended for Automake's configure 
script to use to check if the installed Autoconf has high-resolution 
timestamp support.  It turned out that the test I wrote can give a false 
positive, as some versions of other dependencies of Autom4te::FileUtils 
/also/ use Time::HiRes, causing the test to correctly report that 
Time::HiRes was loaded, but Autom4te::FileUtils nonetheless does not 
actually use it.  The test could probably be improved to fix the false 
positives, but that would be getting into deep magic in Perl that might 
not be fully reliable across Perl versions.  (It would be necessary to 
determine if (a) Time::HiRes::stat exists and (b) 
Autom4te::FileUtils::stat is an alias to it.  Having configure build a 
special XSUB just to check this is well into "ridiculous" territory.)


As such, the Automake maintainers replaced this particular test with a 
simpler test that just locates Autom4te/FileUtils.pm and greps it for 
"Time::HiRes", thus the error message you received, which initially had 
me confused because the test I proposed cannot produce that message as 
it does not use grep.


An important bit of context to keep in mind is that I am not certain 
that timestamp resolution is still a problem outside of the Automake and 
Autoconf testsuites, since Autoconf and Automake now require cache files 
to actually be newer than their sources and consider the cache files 
stale if the timestamps are equal.  This is a problem for the testsuite 
because the testsuite is supposed to actually exercise the caching 
mechanism, and higher-resolution timestamps can significantly reduce the 
amount of time required to run the tests by reducing the delays needed 
to ensure the caches will be valid.



-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Zack Weinberg wrote:

On Sat, Dec 2, 2023, at 7:33 PM, Jacob Bachmeyer wrote:
  

Zack Weinberg wrote:


Would it help if we added a command line option to autom4te that made
it report whether it thought it could use high resolution timestamps?
Versions of autom4te that didn't recognize this option should be
conservatively assumed not to support them.
  

Why not just add that information to the --version message?  Add a
"(HiRes)" tag somewhere if Time::HiRes is available?



Either way is no problem from my end, but it would be more work
for automake (parsing --version output, instead of just checking the
exit status of autom4te --assert-high-resolution-timestamp-support)
Karl, do you have a preference here?  I can make whatever you decide
on happen, in the next couple of days.
  


There would not need to be much parsing, just "automake --version | grep 
HiRes" in that case, or "case `automake --version` in *HiRes*) ...;; 
easc" to avoid running grep if you want.


-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Zack Weinberg wrote:

On Sat, Dec 2, 2023, at 6:37 PM, Karl Berry wrote:
  
The best way to check if high-resolution 
timestamps are available to autom4te is to have perl load 
Autom4te::FileUtils and check if that also loaded Time::HiRes.


The problem with that turned out to be that Time::HiRes got loaded from
other system modules, resulting in the test thinking that autom4te used
it when that wasn't actually the case. That's what happened in practice
with your patch.



Would it help if we added a command line option to autom4te that made it report 
whether it thought it could use high resolution timestamps? Versions of 
autom4te that didn't recognize this option should be conservatively assumed not 
to support them.
  


Why not just add that information to the --version message?  Add a 
"(HiRes)" tag somewhere if Time::HiRes is available?  All versions that 
know to check if Time::HiRes is loaded will also know how to use it, 
unlike the earlier test.



(Of course there's the additional wrinkle that whether high resolution 
timestamps *work* depends on what filesystem autom4te.cache is stored in, but 
that's even harder to probe... one problem at a time?)


Yes; even standard-resolution timestamps might not be "all there" with 
FAT and its infamous 2-second timestamp resolution.


Is this actually still a problem (other than for ensuring the cache is 
used in the testsuite) after Bogdan's patches to require that cache 
files be strictly newer than their source files?



-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Mike Frysinger wrote:

On 06 Apr 2023 21:29, Jacob Bachmeyer wrote:
  

Karl Berry wrote:


jb> a more thorough test would locate the autom4te script and grep it
for the perllibdir that was substituted when autoconf was
configured.

I guess that would work.
  

Challenge accepted.  Here's a refined version:  (lines \-folded for email)

if $PERL -I${autom4te_perllibdir:-$(sed -n \
  '/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}' \
<$(command -v autom4te))} -MAutom4te::FileUtils \
 -e 'exit defined $INC{q[Time/HiRes.pm]} ? 0 : 1'; then
   # autom4te uses Time::HiRes
else
   # autom4te does not use Time::HiRes
fi



this doesn't work on systems that wrap `autom4te`.  Gentoo for example wraps
all autoconf & automake scripts to support parallel installs of different
versions.  this way we can easily have access to every autoconf version.  we
got this idea from Mandrake, so we aren't the only ones ;).
  


If you install a wrapper script, (instead of, for example, making 
autom4te, etc. easily-repointable symlinks), then you must also set 
autom4te_perllibdir in the environment to the appropriate directory when 
building autoconf/automake.  This (with the Gentoo-specific knowledge of 
where the active autom4te is actually located) should be easy to add to 
the ebuild.


If autom4te_perllibdir is set in the environment, its value will be used 
instead of extracting that information from the autom4te script.



[...]

seems like the only reliable option is to invoke autom4te.
am_autom4te_ver=`$AUTOM4TE --version | sed -n '1{s:.*) ::;p}'
AS_CASE([$am_autom4te_ver],
... do the matching ...

what is the first autoconf release that has the fix ?
  


The problem with testing autoconf versions for this is that Time::HiRes 
is an *optional* module in Perl.  It was available from CPAN before it 
was bundled with Perl, and distributions technically can *unbundle* it 
from later Perl releases if they want.  The only reliable way to know if 
Time::HiRes is available (without effectively reimplementing Perl's 
module search) is to try to load it.  Autom4te now (correctly) uses 
Time::HiRes if it is available and falls back to Perl builtins if not, 
for any version of Perl.  The best way to check if high-resolution 
timestamps are available to autom4te is to have perl load 
Autom4te::FileUtils and check if that also loaded Time::HiRes.



-- Jacob

Re: Getting long SOURCES lines with subdirs shorter

[Jacob Bachmeyer]

Jan Engelhardt wrote:

Given

a_SOURCES = aprog/main.c aprog/foo.c aprog/bar.c aprog/baz.c ...

The more source files there are to be listed, the longer that line gets, 
the bigger the Makefile.am fragment becomes, etc. I am thinking about 
how to cut that repetition down. Current automake likely won't have 
anything in store already, so I'm thinking of editing automake and 
targeting a future automake release.
  


While this does not reduce the repetition, Automake allows 
backslash-continuation on these lines.  DejaGnu uses it to list files 
one per line in some places; see 
http://git.savannah.gnu.org/cgit/dejagnu.git/tree/Makefile.am>.


-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Karl Berry wrote:

Hi Jacob,

The guess was the two most probable locations:  /usr/share/autoconf and 
/usr/local/share/autoconf.


Wouldn't have worked on my own system :).

Challenge accepted.  


Thanks!

if $PERL -I${autom4te_perllibdir:-$(sed -n \
  '/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}' \
<$(command -v autom4te))} -MAutom4te::FileUtils \
 -e 'exit defined $INC{q[Time/HiRes.pm]} ? 0 : 1'; then
   # autom4te uses Time::HiRes

unfortunately we are highly restricted in what we can use in basic
automake/conf shell code (as opposed to in the tests).  Neither the
"command" command nor $(...) syntax can be used.
  


Are you sure about that?  I got a fair bit of pushback on removing 
$(...) from config.guess (where it actually is a problem because 
config.guess is supposed to identify a variety of pre-POSIX systems and 
can be run independently of configure) on the grounds that Autoconf 
locates a POSIX shell and uses it for the bulk of configure (and the 
auxiliary scripts like config.guess).  Of course, Autoconf's "find a 
POSIX shell" logic does not help DejaGnu, which installs a copy of 
config.guess and runs it with /bin/sh according to its #! line...



For the former, I think there's an autoconf/make macro to look up a
program name along PATH?


From a quick glance at the manual, that would be 
AC_PATH_PROG([AUTOM4TE], [autom4te]).



[...]
  
Would you be up for tweaking the check to use such

least-common-denominator shell stuff?
  


Let's try:

AC_PATH_PROG([AUTOM4TE], [autom4te])
if test x$autom4te_perllibdir = x; then
 autom4te_perllibdir=`sed -n \
   '/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}' <$AUTOM4TE`
fi
if $PERL -I$autom4te_perllibdir -MAutom4te::FileUtils \
-e 'exit defined $INC{q[Time/HiRes.pm]} ? 0 : 1'; then
 ... 



The backslash-newline in the sed command was added as a precaution 
against line-wrap in email; the line could be combined.



Ordinarily Perl could not be used either, but since Automake is written
in Perl, I don't see a problem with doing so here. (If the system
doesn't have Perl, Automake won't get far.)


If the system lacks Perl, autom4te will not work either.  The proposed 
test uses Perl to determine a characteristic of a program that is 
written in Perl.  :-)



Not sure if $PERL is already
defined by the time at which this would be run, but it should be
possible to arrange with an ac prerequisite if needed.
  


That should be easy enough to rearrange, since this check must come 
/after/ the autoconf version check---the pattern is only valid since 
autoconf-2.52f, but Automake requires autoconf-2.65 or later.



-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Karl Berry wrote:
jb> The test also guesses the location of autoconf's Perl libraries; 


I'm skeptical that any "guessing" of library locations would be reliable
enough.
  


The guess was the two most probable locations:  /usr/share/autoconf and 
/usr/local/share/autoconf.



jb> a more thorough test would locate the autom4te script and grep it
for the perllibdir that was substituted when autoconf was
configured.

I guess that would work.


Challenge accepted.  Here's a refined version:  (lines \-folded for email)

if $PERL -I${autom4te_perllibdir:-$(sed -n \
 '/autom4te_perllibdir/{s/^.*|| //;s/;$//;s/^.//;s/.$//;p;q}' \
   <$(command -v autom4te))} -MAutom4te::FileUtils \
-e 'exit defined $INC{q[Time/HiRes.pm]} ? 0 : 1'; then
  # autom4te uses Time::HiRes
else
  # autom4te does not use Time::HiRes
fi


This version matches a patten that was introduced in commit 
c737451f8c17afdb477ad0fe72f534ea837e001e on 2001-09-13 preceding 
autoconf-2.52f, and Automake currently requires autoconf-2.65 or later, 
so this should work.


Getting the single quotes away from the value without directly 
mentioning them is the purpose of the "s/^.//;s/.$//;" part of the sed 
command.  Wrapping it as "$(eval echo $(sed ...))" would have been 
another option to have the shell strip the single quotes.



Automake and autoconf are not two independent tools. Automake completely
relies on autoconf.

It's not for me to hand down any final pronouncements, but personally I
feel strongly that the tests should not paper over this problem by
changing the way tests work in general. With rm -rf of the cache, or
autoconf -f, etc. That is not what users do, so that's not what the
tests should do, either. Such global changes could have all kinds of
other unknown/undesirable effects on the tests.

In contrast to setting the sleep value "as appropriate", which is what
is/should be already done, so changing the conditions under which it is
set is unlikely to cause any unforeseen additional problems.
  


While potentially compromising the real-world validity of the testsuite 
is a legitimate concern, the fact that Automake depends on Autoconf does 
not preclude the Automake testsuite from working around Autoconf 
limitations in order to accurately test /Automake/.



-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Bogdan wrote:
Jacob Bachmeyer , Mon Apr 03 2023 06:16:53 
GMT+0200 (Central European Summer Time)

Karl Berry wrote:

[...]
   What can we do about this?

As for automake: can we (you :) somehow modify the computation of the
sleep value to determine if autom4te can handle the HiRes testing or 
not

(i.e., has the patch installed)? And then use the longer sleep in
automake testing if needed.


If you can locate Autom4te::FileUtils, grepping it for "Time::HiRes" 
will tell you if autom4te supports sub-second timestamps, but then 
you need more checks to validate that the filesystem actually has 
sub-second timestamps.


A simple check:

if $PERL -I${autom4te_perllibdir:-/usr/share/autoconf} 
-I/usr/local/share/autoconf \
-MAutom4te::FileUtils -e 'exit defined $INC{q[Time/HiRes.pm]} 
? 0 : 1'; then

# autom4te uses Time::HiRes
else
# autom4te does not use Time::HiRes
fi

This method also has the advantage of implicitly also checking that 
$PERL has Time::HiRes installed by determining if loading 
Autom4te::FileUtils causes Time::HiRes to be loaded.  (In other 
words, this will give the correct answer on Perl 5.6 if Time::HiRes 
was installed from CPAN or on later Perls if a distribution packager 
has unbundled Time::HiRes and the user has not installed its package.)



 Nice. The 0 and 1 may not be portable to each OS in the Universe (see 
EXIT_SUCCESS and EXIT_FAILURE in exit(3)), but should be good/portable 
enough for our goals. Or maybe some other simple solution.


Generally, "exit 0" reports success to the shell and any other exit 
value is taken as false.  I am unsure if POSIX actually requires that, 
however.


 As I understand, this could even be used to actually call the sub 
which checks the timestamps, so we'd have a read-to-use test. Only a 
matter of where to put it... Is there some code that runs *before* all 
tests that could set some environment variable passed to the tests, 
create a file, or whatever?


The intended implication was that that test would go in configure.

Verifying that the filesystem actually /has/ subsecond timestamps is a 
separate issue; that test only detects whether autom4te will use 
subsecond timestamps /if/ they are available.


The test also guesses the location of autoconf's Perl libraries; a more 
thorough test would locate the autom4te script and grep it for the 
perllibdir that was substituted when autoconf was configured.



-- Jacob

Re: rhel8 test failure confirmation?

[Jacob Bachmeyer]

Karl Berry wrote:

[...]
   What can we do about this?

As for automake: can we (you :) somehow modify the computation of the
sleep value to determine if autom4te can handle the HiRes testing or not
(i.e., has the patch installed)? And then use the longer sleep in
automake testing if needed.
  


If you can locate Autom4te::FileUtils, grepping it for "Time::HiRes" 
will tell you if autom4te supports sub-second timestamps, but then you 
need more checks to validate that the filesystem actually has sub-second 
timestamps.


A simple check:

if $PERL -I${autom4te_perllibdir:-/usr/share/autoconf} 
-I/usr/local/share/autoconf \
   -MAutom4te::FileUtils -e 'exit defined $INC{q[Time/HiRes.pm]} ? 
0 : 1'; then

   # autom4te uses Time::HiRes
else
   # autom4te does not use Time::HiRes
fi

This method also has the advantage of implicitly also checking that 
$PERL has Time::HiRes installed by determining if loading 
Autom4te::FileUtils causes Time::HiRes to be loaded.  (In other words, 
this will give the correct answer on Perl 5.6 if Time::HiRes was 
installed from CPAN or on later Perls if a distribution packager has 
unbundled Time::HiRes and the user has not installed its package.)



[...]
It seems to me that using autoconf -f or similar is papering over the
problem, so that the tests are no longer testing the normal behavior.
Which does not sound desirable.


The Automake testsuite is supposed to test Automake, not Autoconf, so 
working around Autoconf issues is appropriate.  In this case, if always 
using "autoconf -f" allows us to eliminate the sleeps entirely (and does 
not expand the running time of Autoconf too much), we should do that, at 
least in my view.



-- Jacob

Re: rhel8 test failure confirmation? [PATCH for problem affecting Automake testsuite]

[Jacob Bachmeyer]

A quick introduction to the situation for the Autoconf list:

The Automake maintainers have encountered a bizarre issue with sporadic 
random test failures, seemingly due to "disk writes not taking effect" 
(as Karl Berry mentioned when starting the thread).  Bogdan appears to 
have traced the issue to autom4te caching and offered a patch.  I have 
attached a copy of Bogdan's patch.


Bogdan's patch is a subtle change:  the cache is now considered stale 
unless it is /newer/ than the source files, rather than being considered 
stale only if the source files are newer.  In short, this patch causes 
the cache to be considered stale if its timestamp /matches/ the source 
file, while it is currently considered valid if the timestamps match.  I 
am forwarding the patch to the Autoconf list now because I concur with 
the change, noting that Time:HiRes is also limited by the underlying 
filesystem and therefore is not a "magic bullet" solution.  Assuming the 
cache files are stale unless proven otherwise is therefore correct.


Note again that this is _Bogdan's_ patch I am forwarding unchanged.  I 
did not write it (but I agree with it).


[further comments inline below]

Bogdan wrote:
Bogdan , Sun Mar 05 2023 22:31:55 GMT+0100 (Central 
European Standard Time)
Karl Berry , Sat Mar 04 2023 00:00:56 GMT+0100 
(Central European Standard Time)
 Note that 'config.h' is older (4 seconds) than './configure', 
which

 shouldn't be the case as it should get updated with new values.

Indeed. That is the same sort of thing as I was observing with nodef.
But what (at any level) could be causing that to happen?
Files just aren't getting updated as they should be.

I haven't yet tried older releases of automake to see if their tests
succeed on the systems that are failing now. That's next on my list.


[...]


  Another tip, maybe: cache again. When I compare which files are 
newer than the only trace file I get in the failing 'backcompat2' 
test ('autom4te.cache/traces.0'), I see that 'configure.ac' is older 
than this file in the succeeding run, but it's newer in the failing 
run. This could explain why 'configure' doesn't get updated to put 
new values in config.h (in my case) - 'autom4te' thinks it's up-to-date.

  The root cause may be in 'autom4te', sub 'up_to_date':

   # The youngest of the cache files must be older than the oldest of
   # the dependencies.
   # FIXME: These timestamps have only 1-second resolution.
   # Time::HiRes fixes this, but assumes Perl 5.8 or later.

(lines 913-916 in my version).


This comment Bogdan cites is not correct:  Time::HiRes could be 
installed from CPAN on Perls older than 5.8, and might be missing from a 
5.8 or later installation if the distribution packager separated it into 
another package.  Nor is Time::HiRes guaranteed to fix the issue; the 
infamous example is the FAT filesystem, where timestamps only have 
2-second resolution.  Either way, Time::HiRes is now used if available, 
so this "FIXME" is fixed now.  :-)


  Perhaps 'configure.ac' in the case that fails is created "not late 
enough" (still within 1 second) when compared to the cache, and the 
cached values are taken, generating the old version of 'configure' 
which, in turn, generates old versions of the output files.


  Still a guess, but maybe a bit more probable now.

  Does it work when you add '-f' to '$AUTOCONF'? It does for me - 
again, about 20 sequential runs of the same set of tests and about 5 
parallel with 4 threads. Zero failures.
  I'd probably get the same result if I did a 'rm -fr autom4te.cache' 
before each '$AUTOCONF' invocation.

[...]

  More input (or noise):

1) The t/backcompat2.sh test (the only test which fails for me) is a 
test which modifies configure.ac and calls $AUTOCONF several times.


2) Autom4te (part of Autoconf) has a 1-second resolution in checking 
if the input files are newer than the cache.


Maybe.  That comment could be wrong; the actual "sub mtime" is in 
Autom4te::FileUtils.  Does your version of that module use Time::HiRes?  
Git indicates that use of Time::HiRes was added to Autoconf at commit 
3a9802d60156809c139e9b4620bf04917e143ee2 which is between the 2.72a and 
2.72c snapshot tags.


3) Thus, a sequence: 'autoconf' + quickly modify configure.ac + 
quickly run 'autoconf' may cause autom4te to use the old values from 
the cache instead of processing the new configure.ac. "Quickly" means 
within the same second.


It might be broader than that if your version is already using 
Time::HiRes.  If so, what filesystems are involved?  I could see a 
possible bug where multiple writes get the same mtime if they get 
flushed to disk together.  Time::HiRes will not help if this happens; 
your patch will work around such a bug.


4) I ran the provided list of tests (t/backcompat2.sh, 
t/backcompat3.sh, t/get-sysconf.sh, t/lex-depend.sh, t/nodef.sh, 
t/remake-aclocal-version-mismatch.sh, t/subdir-add2-pr46.sh, 
t/testsuite-summary-reference-log.sh) in batches of 

[bug#61240] [PATCH 2/2] Gracefully degrade if Time::HiRes is not available

[Jacob Bachmeyer]

Paul Eggert wrote:

On 2023-03-28 08:21, Warren Young wrote:
Surely Solaris 10 (shipped in 2005) is relegated to the role of 
porting target, getting nothing but a “dist” tarball?


Good point. I'll cc this to Dagobert Michelsen, who maintains the 
Autoconf port for Solaris 10. Dagobert, are people still running 
Autoconf on Solaris 10? If not, worrying about porting to older Perl 
should be even lower priority than it already is.


For context, this email is about GNU bug 61240:

https://bugs.gnu.org/61240

and the fallout that the latest Autoconf release candidate does not 
run on the old version of Perl installed by default on Solaris 10. See:


https://lists.gnu.org/r/autoconf/2023-03/msg00030.html

If OpenCSW already installs a new-enough Perl version then this 
Autoconf business shouldn't be a real issue. On the other hand if it's 
really trivial to keep Autoconf portable to older Perl, and if Jacob 
Bachmeyer or some other Perl expert can vouch for the change, we might 
as well put it in.


I have since checked and it appears that the same patch I submitted for 
Automake on this bug should also work for Autoconf, applied atop commit 
3a9802d60156809c139e9b4620bf04917e143ee2 (also back out the change to 
"use 5.008" in that commit, which was bogus for the same reasons I have 
previously explained for the Perl version requirement bump in 
Automake).  Autoconf would also do well to revert commit 
61901a1a14fd50c03cfb1529d091554376fef286 and/or possibly split 
m4/perl-time-hires.m4 out of that commit and remove the "use 5.010;" 
line from it; the correct test is for a perl that has Time::HiRes::stat, 
not for any specific version of perl.  I find this particularly grating 
in Autoconf, where the entire point is supposed to be "test for 
features, not versions".  :-/



-- Jacob

[bug#61240] [PATCH 2/2] Gracefully degrade if Time::HiRes is not available

[Jacob Bachmeyer]

[expanding CC list to cover later message]

Paul Eggert wrote:

On 2023-02-07 21:29, Jacob Bachmeyer wrote:

use Exporter;
-use Time::HiRes qw(stat);
use IO::File;

+# use sub-second resolution timestamps if available,
+# carry on with one-second resolution timestamps if that is all we have
+BEGIN { eval { require Time::HiRes; import Time::HiRes qw(stat) } }


Thanks for looking into this. Sorry about the long delay.


I was about to send another ping this weekend.

My memory was jogged by the recent Autoconf release candidate, which 
requires Perl 5.10 for what I think is the same reason 
<https://lists.gnu.org/r/autoconf/2023-03/msg00020.html>. This release 
candidate didn't build on my Solaris 10 server because Solaris 10 has 
only Perl 5.8.4. Of course I can work around this by also installing a 
recent Perl but that is a bit of a pain. I'll cc this email to 
autoc...@gnu.org to give them a heads-up about 
<https://bugs.gnu.org/61240>.


It'd be nice (though not crucial) if we could get to the bottom of 
this for Automake and to sync the result to Autoconf before the new 
Autoconf release comes out, so that Solaris 10 users of the new 
Autoconf need to install only recent GNU M4, and not also a recent Perl.


To get back to the proposed patch quoted above:

Why change from "use Time::HiRes qw(stat);" to "require Time::HiRes; 
import Time::HiRes qw(stat)"? (Again, please bear in mind that my Perl 
is quite rusty.)


In Perl, "use MODULE LIST;" is shorthand for "BEGIN { require Module; 
import Module LIST; }".  The patch must expand that shorthand in order 
to insert an "eval BLOCK" to catch the error "require" will throw if 
Time::HiRes is not available.  In this case, we do not worry about 
actually handling the error, since the fallback is to use the stat 
builtin instead of replacing it with Time::HiRes::stat.


Putting them close together, we have three forms:

(1) use Time::HiRes qw(stat);
(2) BEGIN { require Time::HiRes; import Time::HiRes qw(stat); }
(3) BEGIN { eval { require Time::HiRes; import Time::HiRes qw(stat); } }

According to the Perl manual ("use" in perlfunc), (1) and (2) are 
"exactly equivalent" forms, with a minor syntactic difference not 
relevant here.  (Form (2) is slightly more general.)  The patch replaces 
(1) with (3), which allows the error that occurs if Time::HiRes cannot 
be loaded to be ignored.  Notably, Time::HiRes could be installed from 
CPAN before it became a core module, so this patch also allows 
sub-second timestamps with Perl 5.6 if Time::HiRes has been installed.


Simply saying "eval { use Time::HiRes qw(stat) };" will not work because 
it would expand to "eval { BEGIN { require Time::HiRes; ...} };" which 
would execute the "require" while compiling the "eval" block, and thus 
fail to catch the error if Time::HiRes is not available because the 
error is thrown before the "eval" is actually in force.  Conversely, 
"eval 'use Time::HiRes qw(stat)';" would be too *late*:  the rest of the 
program would have already been compiled to use the stat builtin.  The 
import must be executed before the rest of the program is compiled, and 
an eval must be in force when it is executed to catch the error if 
Time::HiRes is not available.  This requires expanding "use" to its 
equivalent "BEGIN { require ... }" in order to put the "eval" in the 
right place.


The code formerly had "use File::stat;" before it changed to "use 
Time::HiRes qw(stat);". Why doesn't the proposed patch need to fall 
back to "use File::stat;" on older Perls lacking Time::HiRes?


File::stat is a convenience wrapper around the stat builtin that 
modifies it to return an object instead of the 13-element list Perl's 
core stat produces.  File::stat and Time::HiRes::stat are incompatible, 
so the program had to be modified to remove the use of File::stat before 
it could use Time::HiRes::stat.


The reason that the fallback is to do nothing if requiring Time::HiRes 
and importing Time::HiRes::stat fails is that Time::HiRes::stat is a 
drop-in replacement for the stat builtin, which remains available if the 
import fails.  So we attempt to import Time::HiRes::stat (which will 
transparently replace the stat builtin if it succeeds) and proceed with 
the stat builtin (that Perl 5 always has) if the import fails.



Thanks again for any advice you can provide.


You are welcome.  Also note patches at bug#61670 and bug#61671 which 
resolve some minor testsuite issues with Perl 5.6.2.



-- Jacob

Re: if vs. ifdef in Makefile.am

[Jacob Bachmeyer]

Bogdan wrote:

[...]
 Probably Nick's suggestion (a new option to ./configure or the 
AC_HEADER_ASSERT macro) would be the most future-proof, but it 
requires running ./configure each time you wish to change the build 
type (which maybe is not a bad idea, it depends).


That would probably be a very good idea, to avoid mixing files built for 
one mode with files built for another.  Even easier:  use separate build 
directories for each type, from a common source directory, like so:


$ : ... starting one directory above the source tree in ./src/ ...
$ (mkdir test-build; cd ./test-build && ../src/configure --enable-assert 
...)
$ (mkdir release-build; cd ./release-build && ../src/configure 
--disable-assert ...)


Now you avoid conflating modules for test and release builds and ending 
up with an executable that you cannot reliably replicate.  A simple flag 
to make is unlikely to be properly recognized as a dependency for all 
objects built.



-- Jacob

[bug#61671] [PATCH] Remove parentheses around test argument lists

[Jacob Bachmeyer]

This works around a parser bug in Perl 5.6.2.

* t/pm/General.pl: Remove parentheses on argument lists.
---
 t/pm/General.pl | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/t/pm/General.pl b/t/pm/General.pl
index bb99fce83..fafedbc46 100644
--- a/t/pm/General.pl
+++ b/t/pm/General.pl
@@ -18,10 +18,10 @@ use Automake::General;
 my $failed = 0;
 
 # Check 'none'.
-my $none_positive = none { $_[0] < 0 } (1, 7, 3, 8, 9);
+my $none_positive = none { $_[0] < 0 } 1, 7, 3, 8, 9;
 $failed = 1 if ($none_positive == 0);
 
-my $none_gt_8 = none { $_[0] >= 8 } (1, 7, 3, 8, 9);
+my $none_gt_8 = none { $_[0] >= 8 } 1, 7, 3, 8, 9;
 $failed = 1 if ($none_gt_8 == 1);
 
 exit $failed;
-- 
2.17.1

[bug#61670] [PATCH] Improve test for blocked signals

[Jacob Bachmeyer]

This fixes ERRORs reported by:
 * t/parallel-tests-interrupt.tap
 * t/self-check-exit.tap
 * t/self-check-is-blocked-signal.tap
 * t/tap-signal.tap

* t/ax/am-test-lib.sh (is_blocked_signal): Revise Perl code
to more closely follow documented interfaces where
available.  This also works around bugs and limitations
of the POSIX module in Perl 5.6.
---
 t/ax/am-test-lib.sh | 15 ---
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/t/ax/am-test-lib.sh b/t/ax/am-test-lib.sh
index bdf699889..bfa1d84b2 100644
--- a/t/ax/am-test-lib.sh
+++ b/t/ax/am-test-lib.sh
@@ -103,13 +103,14 @@ is_blocked_signal ()
   # Use perl, since trying to do this portably in the shell can be
   # very tricky, if not downright impossible.  For reference, see:
   # 
-  if $PERL -w -e '
-use strict;
-use warnings FATAL => "all";
-use POSIX;
-my %oldsigaction = ();
-sigaction('"$1"', 0, \%oldsigaction);
-exit ($oldsigaction{"HANDLER"} eq "IGNORE" ? 0 : 77);
+  if $PERL -Mstrict -Mwarnings=FATAL,all -MPOSIX -Mconstant=SN,"$1" -e '
+my $new = POSIX::SigAction->new(sub {});
+my $old = POSIX::SigAction->new();
+{ no warnings q[uninitialized]; sigaction(SN, $new, $old) }
+my $oldhandler;
+if ($old->can(q[handler])) { $oldhandler = $old->handler }
+else { $oldhandler = $old->{HANDLER} }
+exit ($oldhandler eq "IGNORE" ? 0 : 77);
   '; then
 return 0
   elif test $? -eq 77; then
-- 
2.17.1

Re: Generating missing depfiles by an automake based makefile

[Jacob Bachmeyer]

Dmitry Goncharov wrote:

On Thursday, February 9, 2023, Tom Tromey  wrote:
  

It's been a long time since I worked on automake, but the dependency
tracking in automake is designed not to need to rebuild or pre-build dep
files.  Doing that means invoking the compiler twice, which is slow.
Instead, automake computes dependencies as a side effect of compilation.


The hello.Po example presented above computes depfiles as a side effect of
compilation. Moreover, when hello.Po is absent that makefile compiles
hello.o as a side effect of hello.Po computation. In total there is only
one compilation.
  

What is the scenario where you both end up with an empty depfile and a
compilation that isn't out of date for some other reason?  That seems
like it shouldn't be possible.


When a depfile is missing (for any reason) the current automake makefile
creates a dummy depfile. From that point on the user has to notice that
make is no longer tracking dependencies and their build is incorrect.

I am asking if automake can be enhanced to do something similar to hello.Po
example above, in those cases when make supports that.


If I understand correctly, the problem here is that the depfile is both 
empty and current.  If Automake could set the dummy depfile's mtime to 
some appropriate past timestamp (maybe the Makefile itself?), it would 
appear out-of-date immediately and therefore be remade, also rebuilding 
the corresponding object.


A quick check of the POSIX manual finds that touch(1) accepts the '-r' 
option to name a reference file and can create a file.  Could we simply 
use "touch -r Makefile $DEPFILE" to create depfiles when we need dummies?



-- Jacob

[bug#61240] [PATCH 1/2] revert Perl version requirement bump

[Jacob Bachmeyer]

Paul Eggert wrote:

On 2023-02-05 21:43, Jacob Bachmeyer wrote:
Should the patch be relative to commit 
6d6fc91c472fd84bd71a1b012fa9ab77bd94efea (before the version 
requirement bump) or should it include reverting commit 
4e3744a15c4d8bdb46c11ead2fb56c5f591b714b (the version requirement bump)?


Might as well do it all at once, thanks.


This first patch is simply `git revert 4e3744`.

8<--
From: Jacob Bachmeyer 
Date: Tue, 7 Feb 2023 22:39:29 -0600
Subject: [PATCH 1/2] Revert "maint: require perl 5.010 or later"

This reverts commit 4e3744a15c4d8bdb46c11ead2fb56c5f591b714b.
---
NEWS   |6 +-
bin/aclocal.in |2 +-
bin/automake.in|2 +-
configure.ac   |5 +++--
lib/Automake/ChannelDefs.pm|2 +-
lib/Automake/Channels.pm   |2 +-
lib/Automake/Condition.pm  |2 +-
lib/Automake/Config.in |2 +-
lib/Automake/Configure_ac.pm   |2 +-
lib/Automake/DisjConditions.pm |2 +-
lib/Automake/FileUtils.pm  |2 +-
lib/Automake/General.pm|2 +-
lib/Automake/Getopt.pm |2 +-
lib/Automake/Item.pm   |2 +-
lib/Automake/ItemDef.pm|2 +-
lib/Automake/Language.pm   |2 +-
lib/Automake/Location.pm   |2 +-
lib/Automake/Options.pm|2 +-
lib/Automake/Rule.pm   |2 +-
lib/Automake/RuleDef.pm|2 +-
lib/Automake/VarDef.pm |2 +-
lib/Automake/Variable.pm   |2 +-
lib/Automake/Version.pm|2 +-
lib/Automake/Wrap.pm   |2 +-
lib/Automake/XFile.pm  |2 +-
25 files changed, 27 insertions(+), 30 deletions(-)

diff --git a/NEWS b/NEWS
index cb32564..8cba8b3 100644
--- a/NEWS
+++ b/NEWS
@@ -5,10 +5,6 @@ please see NEWS-2.0 and start following the advice there now.

New in 1.17:

-* Version requirements:
-
-  - Perl 5.10 (2007) or greater is required.
-
* New features added

  - RANLIB may be overridden on a per-target basis.
@@ -40,7 +36,7 @@ New in 1.17:
and -Q is not used, since its support and behavior varies.

  - Emacs Lisp compilations respects silent make output.
-
+  
  - distcleancheck ignores "silly rename" files (.nfs* .smb* .__afs*)

that can show up on network file systems.

diff --git a/bin/aclocal.in b/bin/aclocal.in
index 34c2530..f04cb30 100644
--- a/bin/aclocal.in
+++ b/bin/aclocal.in
@@ -19,7 +19,7 @@
# Written by Tom Tromey , and
# Alexandre Duret-Lutz .

-use 5.010;
+use 5.006;
use strict;
use warnings FATAL => 'all';

diff --git a/bin/automake.in b/bin/automake.in
index afd296a..139d5ad 100644
--- a/bin/automake.in
+++ b/bin/automake.in
@@ -22,7 +22,7 @@

package Automake;

-use 5.010;
+use 5.006;
use strict;
use warnings FATAL => 'all';

diff --git a/configure.ac b/configure.ac
index bf72023..dcf2d95 100644
--- a/configure.ac
+++ b/configure.ac
@@ -73,9 +73,10 @@ if test -z "$PERL"; then
fi
# Save details about the selected perl interpreter in config.log.
AM_RUN_LOG([$PERL --version])
-$PERL -e 'require 5.010;' || {
+$PERL -e 'require 5.006;' || {
   AC_MSG_ERROR(
-[perl 5.10 (2007) or better is required.  If you have several perl versions
+[perl 5.6 or better is required; perl 5.8.2 or better
+is recommended.  If you have several perl versions
installed, select the one Automake should use using
  ./configure PERL=/path/to/perl])
}
diff --git a/lib/Automake/ChannelDefs.pm b/lib/Automake/ChannelDefs.pm
index bfe5ba5..1c43664 100644
--- a/lib/Automake/ChannelDefs.pm
+++ b/lib/Automake/ChannelDefs.pm
@@ -44,7 +44,7 @@ shorthand function to output on specific channels.

=cut

-use 5.010;
+use 5.006;
use strict;
use warnings FATAL => 'all';

diff --git a/lib/Automake/Channels.pm b/lib/Automake/Channels.pm
index 5a36c93..b4563d3 100644
--- a/lib/Automake/Channels.pm
+++ b/lib/Automake/Channels.pm
@@ -66,7 +66,7 @@ etc.) that can also be overridden on a per-message basis.

=cut

-use 5.010;
+use 5.006;
use strict;
use warnings FATAL => 'all';

diff --git a/lib/Automake/Condition.pm b/lib/Automake/Condition.pm
index d1e6811..31ac81d 100644
--- a/lib/Automake/Condition.pm
+++ b/lib/Automake/Condition.pm
@@ -15,7 +15,7 @@

package Automake::Condition;

-use 5.010;
+use 5.006;
use strict;
use warnings FATAL => 'all';

diff --git a/lib/Automake/Config.in b/lib/Automake/Config.in
index 3cc094d..4fc918b 100644
--- a/lib/Automake/Config.in
+++ b/lib/Automake/Config.in
@@ -17,7 +17,7 @@

package Automake::Config;

-use 5.010;
+use 5.006;
use strict;
use warnings FATAL => 'all';

diff --git a/lib/Automake/Configure_ac.pm b/lib/Automake/Configure_ac.pm
index d4751ee..efd428e 100644
--- a/lib/Automake/Configure_ac.pm
+++ b/lib/Automake/Configure_ac.pm
@@ -20,7 +20,7 @@

package Automake::Configure_ac;

-use 5.010;
+use 5.006;
use strict;
use warnings FATAL => 'all';

diff --git a/lib/Automake/DisjConditions.pm b/lib/Automake/DisjConditions.pm
index 7612f60..16540e7 100644
--- a/lib/Automake/DisjCondi

[bug#61240] [PATCH 2/2] Gracefully degrade if Time::HiRes is not available

[Jacob Bachmeyer]

Paul Eggert wrote:

On 2023-02-05 21:43, Jacob Bachmeyer wrote:
Should the patch be relative to commit 
6d6fc91c472fd84bd71a1b012fa9ab77bd94efea (before the version 
requirement bump) or should it include reverting commit 
4e3744a15c4d8bdb46c11ead2fb56c5f591b714b (the version requirement bump)?


Might as well do it all at once, thanks.


This second patch is the change I previously suggested.  As noted in the 
patch, please use my gnu.org address for public attribution.  (I would 
prefer to minimize any indirect advertising of GMail.)


I have found no better ChangeLog entry than the commit message:

   * lib/Automake/FileUtils.pm: Gracefully degrade if Time::HiRes is 
not available.


8<--
From: Jacob Bachmeyer 
Date: Tue, 7 Feb 2023 22:42:59 -0600
Subject: [PATCH 2/2] Gracefully degrade if Time::HiRes is not available

---
lib/Automake/FileUtils.pm |5 -
1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/lib/Automake/FileUtils.pm b/lib/Automake/FileUtils.pm
index 697ff7e..8d0b368 100644
--- a/lib/Automake/FileUtils.pm
+++ b/lib/Automake/FileUtils.pm
@@ -39,9 +39,12 @@ use strict;
use warnings FATAL => 'all';

use Exporter;
-use Time::HiRes qw(stat);
use IO::File;

+# use sub-second resolution timestamps if available,
+# carry on with one-second resolution timestamps if that is all we have
+BEGIN { eval { require Time::HiRes; import Time::HiRes qw(stat) } }
+
use Automake::Channels;
use Automake::ChannelDefs;

--
8<--


-- Jacob

[bug#61240] [PATCH 0/2] restore support for 5.6 and gracefully degrade timestamps

[Jacob Bachmeyer]

Paul Eggert wrote:

On 2023-02-05 21:43, Jacob Bachmeyer wrote:
Should the patch be relative to commit 
6d6fc91c472fd84bd71a1b012fa9ab77bd94efea (before the version 
requirement bump) or should it include reverting commit 
4e3744a15c4d8bdb46c11ead2fb56c5f591b714b (the version requirement bump)?


Might as well do it all at once, thanks.


Two patches, generated with git format-patch as requested, follow under 
separate cover.


I had to rework my local repository, where I had previously rolled back 
to before the version requirement bump instead of reverting it.  Using 
"git diff" confirms no differences between the two local branch tips, so 
I am sending in the patch now while tests continue to run here.  Patch 1 
reverts the version requirement bump; patch 2 allows Automake to 
gracefully degrade if Time::HiRes is not available.


While Automake clearly works correctly under Perl 5.6.2 with these 
patches applied, I am still running down some minor testsuite result 
differences between the system Perl 5.34.1 and perlbrew Perl 5.6.2 on 
the machine I am using for the tests.  Should patches to correct these 
issues be sent under this bug or should they be sent separately to 
automake-patches?


Preliminary analysis suggests that the Perl fragment in 
t/ax/am-test-lib.sh:is_blocked_signal needs a few improvements (to 
improve compatibility in both directions; an accessor method was 
introduced in Perl 5.8 and the code fails under Perl 5.6 due to other 
limitations; [*facepalm*]), and t/pm/General.pl appears to be tickling a 
compiler bug in Perl 5.6.2 that produces a false syntax error, but 
Automake itself is unaffected.  The only other differences are 
t/pm/Condition-t.pl and t/pm/DisjConditions-t.pl, which are skipped due 
to a lack of the required thread support in the older Perl.  The failure 
of is_blocked_signal causes ERRORs in t/parallel-tests-interrupt.tap, 
t/self-check-exit.tap, t/self-check-is-blocked-signal.tap, and 
t/tap-signal.tap.  No other tests FAIL with either Perl.


I also found that t/get-sysconf.sh FAILs if libtool is not installed, 
but configure does not detect and complain about the unavailability of 
libtoolize.



-- Jacob

[bug#61240] improve high-res file timestamp in Automake

[Jacob Bachmeyer]

Paul Eggert wrote:

On 2023-02-05 17:14, Jacob Bachmeyer wrote:

[...]
Looking at the code, commit 01bf65daf6f6627b56fbe78fc436fd877ccd3537 
appears fine, all I am asking is that commit 
4e3744a15c4d8bdb46c11ead2fb56c5f591b714b be reverted.  The current 
Automake Git master should actually work on Perl 5.6 if Time::HiRes 
has been installed, which was possible with 5.6 although it was 
bundled with Perl beginning with the 5.7.3 development release.


Unfortunately the Perl version bump was prompted by evidence in the 
field that without making it clear that bare Perl 5.6 does not 
suffice, Autoconf and Automake fail in ways that are mysterious to 
their users. We can't expect people to install extensions in Perl 5.6 
to work around this problem. We must make things simple and easy for 
installers and users to understand. Particularly since these are old 
Perl versions that Autoconf and Automake users are unlikely to be 
running (people who use bleeding-edge Autoconf and Automake almost 
invariably run recent Perl).


I believe the proper autoconf solution would be for configure to verify 
that "$PERL -MTime::HiRes -e 1" completes successfully (that is, that 
$PERL has the Time::HiRes module available) and fail with a diagnostic 
that Time::HiRes is required if not, perhaps noting that Time::HiRes was 
bundled beginning with Perl 5.8.  (The 5.7 series were development 
releases.)


Note that the real requirement here is not a given Perl version, but the 
Time::HiRes module (solution to that below), so checking the Perl 
version is incorrect here.  Even though Perl 5.8 and later have 
Time::HiRes bundled, it is still a module and it is possible for a 
distribution package builder to separate Time::HiRes from the main perl 
package, so there is no guarantee that Time::HiRes will actually be 
available unless you explicitly test for it.  I doubt that any major 
distribution currently does this with Time::HiRes, but the Perl version 
does not actually imply the availability of any given module.


It would be OK to go back to requiring only 5.6 if we can write 
conditionalized code that works with 5.6 but with lower-res 
timestamps, and quietly switches to higher-res timestamps if available 
and works fine when they are available. Autoconf and Automake should 
not rely on users installing optional Perl packages or dealing with 
Perl's diagnostics when the optional stuff is missing: the code must 
work out of the box with no expertise required.


I believe that is quite easy to do:

   BEGIN { eval { require Time::HiRes; Time::HiRes->import('stat') } }

instead of:

   use Time::HiRes qw(stat);

In Perl, "use Module LIST" is shorthand for "BEGIN { require Module; 
import Module LIST; }", so the above could also be written as "BEGIN { 
eval { require Time::HiRes; import Time::HiRes qw(stat) } }" if you 
wanted, although the direct package method call is a more common style.


If Time::HiRes is available, that will transparently replace stat() with 
Time::HiRes::stat(); if not, processing continues using the basic 
stat(), since require throws an exception immediately if it fails, which 
the eval BLOCK will catch.  This has the advantage of testing for the 
specific feature, and will give high-resolution timestamps even on 5.6 
if Time::HiRes is installed and basic timestamps otherwise.


Since Perl 5.8 and later all have Time::HiRes bundled, the import should 
succeed and high resolution timestamps will be available.  I say 
/should/ because it is possible for distribution packagers to separate 
bundled modules out from their main perl packages, as I mentioned above.


Can you write an Automake patch to do that, and test it on old Perl 
installations? 'git format-patch' form preferred. I don't have access 
to ancient Perl and am a bit squeezed for time, so I'm afraid this 
will need to be done by a Perl expert such as yourself.


The oldest Perl I normally use is a 5.8 installation from source that 
has (bundled) Time::HiRes, so I have perlbrew cooking up a 5.6.2 to test 
the fallback as I write this.


Should the patch be relative to commit 
6d6fc91c472fd84bd71a1b012fa9ab77bd94efea (before the version requirement 
bump) or should it include reverting commit 
4e3744a15c4d8bdb46c11ead2fb56c5f591b714b (the version requirement bump)?



-- Jacob

[bug#61240] improve high-res file timestamp in Automake

[Jacob Bachmeyer]

Paul Eggert wrote:

On 2023-02-04 16:02, Jacob Bachmeyer wrote:
In any case, you will still need to account for the possibility that 
Time::HiRes::stat() might not actually have higher resolution, 
depending on the filesystem.


That's fine. All we want is the exact file timestamp. If the file 
system timestamp resolution is only 2 s, then we want that multiple of 
2. Admittedly we can't get the exact file timestamp on many modern 
file systems since Time::HiRes is precise only to the nearest ~238 ns 
for today's timestamps, but the idea is to get what we easily can.


How often is Perl built to use long doubles these days?  (That was an 
option beginning with Perl 5.6.)


There are also the no-runtime-overhead options of using "eval { use 
Time::HiRes qw(stat) };" which will replace stat() with the hi-res 
version if it is available and continue with the regular stat() 
builtin if not, or "use constant HAVE_Time_HiRes => eval { use 
Time::HiRes };" and a conditional "if (HAVE_Time_HiRes) { ... } else 
{ ... }" as I suggested as an improvement to Mike Frysinger's patch. 


Sorry, I don't remember seeing that suggestion. I guess it was in 
another thread. Could you resend that patch to 61...@debbugs.gnu.org 
and cc me? Preferably a patch against the latest Automake, in "git 
format-patch" format; see 
<https://savannah.gnu.org/git/?group=automake> for how to get 
bleeding-edge Automake. The idea would be to port bleeding-edge 
Automake to Perl < 5.10 when that's easy.


It was bug#60807.  Looking at the patch, I suspect that I will need to 
take a closer look at the code:  the existing code before Mike's patch 
used File::stat, which I suspect may not be compatible with 
Time::HiRes::stat (not compatible in the sense that File::stat's 
override of stat() probably will not use Time::HiRes's override of 
stat()).  If so, then "use Time::HiRes qw(stat);" will not do what you 
want because either stat() will either cease to return an object or will 
not have subsecond resolution.


Looking at the code, commit 01bf65daf6f6627b56fbe78fc436fd877ccd3537 
appears fine, all I am asking is that commit 
4e3744a15c4d8bdb46c11ead2fb56c5f591b714b be reverted.  The current 
Automake Git master should actually work on Perl 5.6 if Time::HiRes has 
been installed, which was possible with 5.6 although it was bundled with 
Perl beginning with the 5.7.3 development release.


In other words, bleeding-edge Automake is already ported to Perl < 5.10, 
in fact it should run on 5.8 out-of-the-box and 5.6 if Time::HiRes has 
been installed from CPAN (or perhaps BackPAN today, but Time::HiRes was 
on CPAN when 5.6 was current).


Please bear in mind that I stopped coding in Perl 30 years ago and so 
am a bit rusty.


Yes, very---30 years ago would have been before Perl 5.6!  Much of 
modern Perl did not exist then...



-- Jacob

[bug#61240] improve high-res file timestamp in Automake

[Jacob Bachmeyer]

Paul Eggert wrote:

On 2023-02-03 18:27, Jacob Bachmeyer wrote:

Where are you actually using a 5.10 feature?


Where lib/Automake/FileUtils.pm says "use Time::HiRes qw(stat);". This 
does not work with Perl 5.6.


Time::HiRes is (perhaps was) installable from CPAN and is definitely 
/not/ a 5.10 feature.  I have a Perl 5.8 with it installed, and while my 
memories that far back are a bit fuzzy, I seem to remember installing 
Time::HiRes on a Perl 5.6 installation some years ago.  Things like 
"our" variables, PerlIO-by-default, and the defined-or operator are Perl 
features (those in 5.6, 5.8, and 5.10 if I remember correctly), modules 
are (with rare exceptions) not Perl features.


The correct solution if you do not want to provide for the case where 
Time::HiRes is not available is to simply "use Time::HiRes qw(stat);" 
and *let* *that* *fail* if Time::HiRes is not available or cannot export 
stat().  Time::HiRes was on CPAN long before it was bundled with Perl, 
so the Perl version does /not/ tell you if it is available.  (If 
Automake::FileUtils is not immediately loaded, put "use Time::HiRes;" 
somewhere that is.  That will be enough to ensure that Time::HiRes is 
available.)



For why we bumped the version to 5.10, please see:

https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=61901a1a14fd50c03cfb1529d091554376fef286 



Please do not arbitrarily bump version requirements just to bump 
version requirements.


That's not what was done here. The abovementioned URL says version 
requirements were bumped from 5.6 to 5.10 because the feature is not 
present in 5.6 (2000), is present in 5.10 (2007), and we lacked access 
to the museum pieces in the middle. If you are sure that a version 
number lower than 5.10 will do, please let us know.


This was arbitrary because Time::HiRes is not actually dependent on the 
Perl version in that way.  (You should also be able to get access to the 
museum pieces in the middle fairly easily using perlbrew.)


I am fairly sure that Time::HiRes could be installed from CPAN on at 
least Perl 5.6 and later, and possibly even on Perls older than 5.6.  
(To add it to a 5.6 installation today may require retrieving it from 
BackPAN.)  I definitely know that it can be installed on 5.8, as I still 
have a Perl 5.8 that has it, and "use Time::HiRes qw(stat);" succeeds in 
that Perl.  If the requirement is Time::HiRes, then the proper course of 
action is to "require Time::HiRes;" rather than requiring some version 
of Perl that you know shipped it as a core module.  Doing the latter 
/is/ arbitrarily bumping the version requirement.


In fact, according to my copy of Module::CoreList (or, to be precise, 
its corelist(1) command-line frontend), Time::HiRes was first bundled 
with Perl 5.7.3.  Note that I ran that query against a Perl 5.34 
installation; the Perl community takes backwards compatibility like this 
very seriously.  Note also that I said "bundled"; it could be installed 
from CPAN in older Perls, so requiring 5.8 (since 5.7 was a development 
series) is /not/ correct here.


There are also the no-runtime-overhead options of using "eval { use 
Time::HiRes qw(stat) };" which will replace stat() with the hi-res 
version if it is available and continue with the regular stat() builtin 
if not, or "use constant HAVE_Time_HiRes => eval { use Time::HiRes };" 
and a conditional "if (HAVE_Time_HiRes) { ... } else { ... }" as I 
suggested as an improvement to Mike Frysinger's patch.


In any case, you will still need to account for the possibility that 
Time::HiRes::stat() might not actually have higher resolution, depending 
on the filesystem.  FAT, in particular, is notorious for its 2-second 
timestamp resolution.  (This issue is actually mentioned in the 
Time::HiRes::stat documentation.)



-- Jacob

[bug#61240] improve high-res file timestamp in Automake

[Jacob Bachmeyer]

Paul Eggert wrote:
I installed the attached to port a FileUtils.pm patch back from 
Autoconf into Automake. I wish Perl supported file timestamps with 
nanosecond resolution, but apparently not, so this is the best we 
could do easily.


Although this bumps the required Perl version from 5.6 (2000) to 5.10 
(2007), I don't think that's a problem nowadays.


Where are you actually using a 5.10 feature?  Please do not arbitrarily 
bump version requirements just to bump version requirements.



-- Jacob

[bug#60807] [PATCH v2] tests: reuse am_cv_filesystem_timestamp_resolution

[Jacob Bachmeyer]

Mike Frysinger wrote:

On 14 Jan 2023 21:43, Jacob Bachmeyer wrote:
  
[...]
You could also exploit that || short-circuits in the shell and replace 
the "if" block with " $PERL ... || sleep='sleep ''2' ".  This allows you 
to directly execute a command on a false result and (I think) it is 
portable, too.  (I half-expect someone to correct me on that along the 
lines of "the shell on Obscurix has a bug where || implicitly uses a 
subshell".)



personally i find (ab)use of `||` and `&&` tends to lead to unmaintainable code.
i.e. it tends to produce write-once-read-never code akin to most perl.  so if
the construct has been in use already and isn't causing issues, i'd use it.


In Perl at least, use of the "word forms" of those operators for flow 
control is considered idiomatic Perl, most often seen in the form "open 
... or die ...;" and indeed in another of your patches as "stat ... or 
fatal ...".  :-)



-- Jacob

[bug#60807] [PATCH v2] tests: reuse am_cv_filesystem_timestamp_resolution

[Jacob Bachmeyer]

Mike Frysinger wrote:

Rather than assume such coarse delays, re-use existing logic for
probing the current filesystem resolution.  This speeds up the
testsuite significantly.  On my system, it speeds -j1 up quite a
lot -- by ~30%.  While I didn't gather many samples to produce a
statistically significant distribution, my runs seem to be fairly
consistent with the values below with deviations of <1 minute.

[...]
diff --git a/t/aclocal-no-force.sh b/t/aclocal-no-force.sh
index 3e0c04d12f18..2e139d75cf74 100644
--- a/t/aclocal-no-force.sh
+++ b/t/aclocal-no-force.sh
@@ -19,6 +19,18 @@
 
 . test-init.sh
 
+# Automake relies on high resolution timestamps in perl.  If support isn't

+# available (see lib/Automake/FileUtils.pm), then fallback to coarse sleeps.
+# The creative quoting is to avoid spuriously triggering a failure in
+# the maintainer checks.
+case ${sleep_delay} in
+0*)
+  if ! $PERL -e 'use Time::HiRes' 2>/dev/null; then
+sleep='sleep ''2'
+  fi
+  ;;
+esac
+
 cat >> configure.ac << 'END'
 SOME_DEFS
 AC_CONFIG_FILES([sub/Makefile])
  


I seem to remember being told that "if !" is non-portable.  Is there 
some other mechanism that ensures this is always run with Bash or might 
"if $PERL ... ; then :; else" be a better option for that line?


Also, you could write that Perl command as "$PERL -MTime::HiRes -e 1 
2>/dev/null" and avoid needing any quotes there, although I suspect this 
is simply a matter of style and the comment refers to the quotes when 
setting $sleep.


You could also exploit that || short-circuits in the shell and replace 
the "if" block with " $PERL ... || sleep='sleep ''2' ".  This allows you 
to directly execute a command on a false result and (I think) it is 
portable, too.  (I half-expect someone to correct me on that along the 
lines of "the shell on Obscurix has a bug where || implicitly uses a 
subshell".)



-- Jacob

[bug#60807] [PATCH 1/2] mtime: use Time::HiRes::stat when available for subsecond resolution

[Jacob Bachmeyer]

Mike Frysinger wrote:

Perl's builtin stat function returns timestamps that have 1 second
resolution.  This can lead automake needlessly regenerating files
because it compares timestamps as "older than or equal to" rather
than only "older than".  This is perfectly reasonable as we have
no way of knowing what file is older if they have the same value
which means we must be pessimistic & assume an upate is required.

However, given modern systems that are quite fast and can easily
generate many files in less than an second, we end up doing a lot
of extra work.

Until Perl gets around to figuring out how to support subsecond
timestamp resolution, optionally import the Time::HiRes module and
use its stat function instead.  If it's available, great, if not,
then we're no worse off than we are today.

Performance-wise, at least by using the testsuite, there doesn't
seem to be any measurable difference.

* lib/Automake/FileUtils.pm: Use Time::HiRes to lookup mtimes on
files if available.
---
 lib/Automake/FileUtils.pm | 21 +
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/lib/Automake/FileUtils.pm b/lib/Automake/FileUtils.pm
index 848ff22d1761..78e0942e9f53 100644
--- a/lib/Automake/FileUtils.pm
+++ b/lib/Automake/FileUtils.pm
@@ -42,6 +42,11 @@ use Exporter;
 use File::stat;
 use IO::File;
 
+# Perl's builtin stat does not provide sub-second resolution.  Use Time::HiRes

+# if it's available instead.  Hopefully one day perl will update.
+# https://github.com/Perl/perl5/issues/17900
+my $have_time_hires = eval { require Time::HiRes; };
+
 use Automake::Channels;
 use Automake::ChannelDefs;
 
@@ -115,10 +120,18 @@ sub mtime ($)

   return 0
 if $file eq '-' || ! -f $file;
 
-  my $stat = stat ($file)

-or fatal "cannot stat $file: $!";
-
-  return $stat->mtime;
+  if ($have_time_hires)
+{
+  my @stat = Time::HiRes::stat ($file)
+   or fatal "cannot stat $file: $!";
+  return $stat[9];
+}
+  else
+{
+  my $stat = stat ($file)
+   or fatal "cannot stat $file: $!";
+  return $stat->mtime;
+}
 }


If you change that variable to a constant, you can eliminate the runtime 
overhead entirely, since Perl optimizes if(1) and if(0) and folds 
constants at compile time.


Something like:

   use constant HAVE_Time_HiRes => eval { require Time::HiRes; };

Then:

   if (HAVE_Time_HiRes)
  ...

If you do this, Perl will inline the block actually used and elide the 
branch at runtime.  This is generally useful for any test that can only 
go one way in a specific run of the program.



-- Jacob

[bug#59989] [PATCH] tests: Fix txinfo-include test for texinfo 7.x

[Jacob Bachmeyer]

Karl Berry wrote:

Hi Frederic,

Texinfo modified its behavior regarding apostrophes, which are now
replaced by UTF-8 right single quotes by default.

Sorry to hear it, but not surprised.

-GNU's Not Unix.
+@verb{.GNU's Not Unix..}

I don't think there's any guarantee that verb quotes will also be
"helpfully" changed in the future. One wouldn't think so, but ...
  


There has to be some way to get a straight quote, and verbatim is (as 
far as I can tell) intended for code examples and similar literal text.  
Note that TeX has long (always?) converted backtick and apostrophe to 
left and right quotes, building up double quotes from `` and '', so this 
change in Texinfo harmonizes Info output with what texinfo.tex has 
always produced.  (There have been a few places in the DejaGnu manual 
where I have converted ASCII double quotes to TeX style because regular 
double quotes look seriously out of place in the PDF version.)



So instead, I changed the test document to avoid apostrophes. Pushed the
below. Hope it flies. --thanks, karl.
  


Since this is a test document, that should not be a problem.


-- Jacob

Re: man_MANS install locations

[Jacob Bachmeyer]

Karl Berry wrote:

Hi Jan,

As for GNU/Linux, what was the rationale to only permit [0-9ln]?

No idea. Maybe just didn't think about "m", or maybe it didn't exist at
that time? Jim, Paul, anyone?

Should automake be relaxed? 


I see no harm in allowing more (any) letters, if that's what you mean.

When running automake on Solaris, placing svcadm.1m into man1 rather
than man1m seems outright wrong.

But is Automake's purpose to reproduce platform-specific behavior, or to
have consistent behavior across platforms?  I think the latter.
  


This would be adapting to platform-specific requirements.  I suspect 
that Solaris man(1) will not look for svcadm.1m in man1 at all but only 
in man1m.



I guess a new option to install *.1m in man1m/, etc., would be ok, if
you want it. If you or anyone can provide a patch, that would be
great. Unfortunately I doubt it's anything I will ever implement myself.
  


Maybe the best answer is to install into an existing directory if one is 
found and otherwise trim the suffix to the "standard" set?



Should the rpmlint check be adjusted to cater to the GNU FHS?

I guess that's a question for the rpmlint people, whoever they are.
I don't see that Automake's default behavior is going to change.

Also, GNU (as an organization) never had anything to do with the FHS,
so far as I know. I don't think the GNU coding standards/maintainer
information have anything to say about this topic ...
  


I seem to remember reading somewhere that /usr is supposed to be a 
symlink to / on the GNU system, so no, GNU is not intended to follow FHS.



-- Jacob

Re: Old .Po file references old directory, how to start fresh?

[Jacob Bachmeyer]

Travis Pressler via Discussion list for automake wrote:

Hi,

I'm learning how to make an autotools project and have created a test project 
to work with. I ran make with a directory `nested` and then deleted it and 
deleted the reference to it in my `Makefile.am`.

Now I'm running ./configure && make​ and I get the following:

*** No rule to make target 'nested/main.c', needed by 'main.o'. Stop.​

How can I run `make` so that it doesn't reference this old nested​ directory?

I was curious if I could find where this reference is, so I did a grep -r 
nested .​ I think the only relevant hit is:

./src/.deps/main.Po:main.o nested/main.c /usr/include/stdc-predef.h 
/usr/include/stdio.h \​


Have you rerun automake to regenerate Makefile.in since changing 
Makefile.am?



-- Jacob

Re: type errors, command length limits, and Awk

[Jacob Bachmeyer]

Mike Frysinger wrote:

On 15 Feb 2022 21:17, Jacob Bachmeyer wrote:
  

Mike Frysinger wrote:


context: https://bugs.gnu.org/53340
  
  

Looking at the highlighted line in the context:



thanks for getting into the weeds with me
  


You are welcome.


  echo "$$py_files" | $(am__pep3147_tweak) | $(am__base_list) | \

It seems that the problem is that am__base_list expects ListOf/File (and 
produces ChunkedListOf/File) but am__pep3147_tweak emits ListOf/Glob.  
This works in the usual case because the shell implicitly converts Glob 
-> ListOf/File and implicitly flattens argument lists, but results in 
the overall command line being longer than expected if the globs expand 
to more filenames than expected, as described there.


It seems that the proper solution to the problem at hand is to have 
am__pep3147_tweak expand globs itself somehow and thus provide 
ListOf/File as am__base_list expects.


Do I misunderstand?  Is there some other use for xargs?



if i did not care about double expansion, this might work.  the pipeline
quoted here handles the arguments correctly (other than whitespace splitting
on the initial input, but that's a much bigger task) before passing them to
the rest of the pipeline.  so the full context:

  echo "$$py_files" | $(am__pep3147_tweak) | $(am__base_list) | \
  while read files; do \
$(am__uninstall_files_from_dir) || st=$$?; \
  done || exit $$?; \
...
am__uninstall_files_from_dir = { \
  test -z "$$files" \
|| { test ! -d "$$dir" && test ! -f "$$dir" && test ! -r "$$dir"; } \
|| { echo " ( cd '$$dir' && rm -f" $$files ")"; \
 $(am__cd) "$$dir" && rm -f $$files; }; \
  }

leveraging xargs would allow me to maintain a single shell expansion.
the pathological situation being:
  bar.py
  __pycache__/
bar.pyc
bar*.pyc
bar**.pyc

py_files="bar.py" which turns into "__pycache__/bar*.pyc" by the pipeline,
and then am__uninstall_files_from_dir will expand it when calling `rm -f`.

if the pipeline expanded the glob, it would be:
  __pycache__/bar.pyc __pycache__/bar*.pyc __pycache__/bar**.pyc
and then when calling rm, those would expand a 2nd time.
  


If we know that there will be _exactly_ one additional shell expansion, 
why not simply filter the glob results through `sed 's/[?*]/\\&/g'` to 
escape potential glob metacharacters before emitting them from 
am__pep3147_tweak?  (Or is that not portable sed?)


Back to the pseudo-type model I used earlier, the difference between 
File and Glob is that Glob contains unescaped glob metacharacters, so 
escaping them should solve the problem, no?  (Or is there another thorn 
nearby?)



[...]

which at this point i've written `xargs -n40`, but not as fast :p.
  


Not as fast, yes, but certainly portable!  :p

The real question would be if it is faster than simply running rm once 
per file.  I would guess probably _so_ on MinGW (bash on Windows, where 
that logic would use shell builtins but running a new process is 
extremely slow) and probably _not_ on an archaic Unix system where 
"test" is not a shell builtin so saving the overhead and just running rm 
once per file would be faster.



automake jumps through some hoops to try and limit the length of generated
command lines, like deleting output objects in a non-recursive build.  it's
not perfect -- it breaks arguments up into 40 at a time (akin to xargs -n40)
and assumes that it won't have 40 paths with long enough names to exceed the
command line length.  it also has some logic where it's deleting paths by
globs, but the process to partition the file list into groups of 40 happens
before the glob is expanded, so there are cases where it's 40 globs that can
expand into many many more files and then exceed the command line length.
  
First, I thought that GNU-ish systems were not supposed to have such 
arbitrary limits,



one person's "arbitrary limits" is another person's "too small limit" :).
i'm most familiar with Linux, so i'll focus on that.

[...]

plus, backing up, Automake can't assume Linux.  so i think we have to
proceed as if there is a command line limit we need to respect.
  


So then the answer to my next question is that it is still an issue, 
even if the GNU system were to allow arguments up to available memory.


and this issue (the context) originated from Gentoo 
GNU/Linux.  Is this a more fundamental bug in Gentoo or still an issue 
because Automake build scripts are supposed to be portable to foreign 
system that do have those limits?



to be clear, what's failing is an Automake test.  it sets the `rm` limit to
an articially low one.  [...]

Gentoo happened to find this error before Automake because Gentoo also found
and fixed a Python 3.5+ problem -- https://bugs.gnu.org/38043.  once we fix
that in A

type errors, command length limits, and Awk (was: portability of xargs)

[Jacob Bachmeyer]

Mike Frysinger wrote:

context: https://bugs.gnu.org/53340
  

Looking at the highlighted line in the context:

>   echo "$$py_files" | $(am__pep3147_tweak) | $(am__base_list) | \
It seems that the problem is that am__base_list expects ListOf/File (and 
produces ChunkedListOf/File) but am__pep3147_tweak emits ListOf/Glob.  
This works in the usual case because the shell implicitly converts Glob 
-> ListOf/File and implicitly flattens argument lists, but results in 
the overall command line being longer than expected if the globs expand 
to more filenames than expected, as described there.


It seems that the proper solution to the problem at hand is to have 
am__pep3147_tweak expand globs itself somehow and thus provide 
ListOf/File as am__base_list expects.


Do I misunderstand?  Is there some other use for xargs?

I note that the current version of standards.texi also allows configure 
and make rules to use awk(1); could that be useful here instead? (see below)



[...]

automake jumps through some hoops to try and limit the length of generated
command lines, like deleting output objects in a non-recursive build.  it's
not perfect -- it breaks arguments up into 40 at a time (akin to xargs -n40)
and assumes that it won't have 40 paths with long enough names to exceed the
command line length.  it also has some logic where it's deleting paths by
globs, but the process to partition the file list into groups of 40 happens
before the glob is expanded, so there are cases where it's 40 globs that can
expand into many many more files and then exceed the command line length.
  


First, I thought that GNU-ish systems were not supposed to have such 
arbitrary limits, and this issue (the context) originated from Gentoo 
GNU/Linux.  Is this a more fundamental bug in Gentoo or still an issue 
because Automake build scripts are supposed to be portable to foreign 
system that do have those limits?


Second, counting files in the list, as you note, does not necessarily 
actually conform to the system limits, while Awk can track both number 
of elements in the list and the length of the list as a string, allowing 
to break the list to meet both command tail length limits (on Windows or 
total size of block to transfer with execve on POSIX) and argument count 
limits (length of argv acceptable to execve on POSIX).


POSIX Awk should be fairly widely available, although at least Solaris 
10 has a non-POSIX awk in /usr/bin and a POSIX awk in /usr/xpg4/bin; I 
found this while working on DejaGnu.  I ended up using this test to 
ensure that "awk" is suitable:


8<--
# The non-POSIX awk in /usr/bin on Solaris 10 fails this test
if echo | "$awkbin" '1 && 1 {exit 0}' > /dev/null 2>&1 ; then
   have_awk=true
else
   have_awk=false
fi
8<--


Another "gotcha" with Solaris 10 /usr/bin/awk is that it will accept 
"--version" as a valid Awk program, so if you use that to test whether 
"awk" is GNU Awk, you must redirect input from /dev/null or it will hang.


Automake may want to do more extensive testing to find a suitable Awk; 
the above went into a script that remains generic when installed and so 
must run its tests every time the user invokes it, so "quick" was a high 
priority.



-- Jacob

Re: portability of xargs

[Jacob Bachmeyer]

Dan Kegel wrote:

Meson is a candidate for such a next-gen config system.  It is in python,
which does not quite qualify as usable during early uplift/bootstrap, but
there are C ports in progress, see e.g. https://sr.ht/~lattis/muon/
  


*Please* do not introduce a dependency on Python; they do not worry much 
about backwards compatibility.  If there is ever a Python 4 with a 3->4 
transition anything like the 2->3 transition, you could end up with 
every past release relying on current Python becoming unbuildable.


Having complex dependencies for creating the build scripts is one thing, 
but needing major packages (like Python) to *use* the build scripts is a 
serious problem for anything below the "user application" tier, 
especially the "base system" tier.



-- Jacob

Re: [PATCH] python: prioritize python 3.x over 2.x

[Jacob Bachmeyer]

Zack Weinberg wrote:

On Wed, Jan 26, 2022, at 10:10 PM, Mike Frysinger wrote:
  

On 26 Jan 2022 10:07, Zack Weinberg wrote:


Please also move the plain "python" command down to the python2
block (right after "python2" would be the best place I think).  Any
system on which "python" runs Python 3 is grossly misconfigured.
  

i don't think that's accurate.  Python PEP-0394 explicitly permits
`python` to be any version, and many distros do just that



Yes, I know.  PEP 394 is wrong.  Its authors grossly underestimated
the volume of Python 2 code that still exists today and, if
accidentally executed with a v3 interpreter, will *silently*
malfunction -- not just crashing, but corrupting data with no warning.
I had to deal with this kind of code all the time in my previous day
job and I'm still quite peeved that the Python devs blew me off about it.

What PEP 394 *should* have said is that the command name "python" and
the #! path /usr/bin/python MUST[rfc2119] be *permanently reserved*
for the Python 2.7 interpreter, and "python3" is the only acceptable
not-specifically-versioned command name for the 3.x interpreter, even
in virtualenvs.

  

distros that remove `python` entirely as part of Python 2 removal
are doing it wrong.



On the contrary, those distros are the only ones doing it right.

Now, nobody listens to me, and I'm painfully aware that "python"
pointing to Python 3 is common, but that doesn't mean Autotools should
make the situation even worse.  Autotools should, on its own motion,
discourage the use of "python".  Thus, my suggested changes.
  


Are Python 2 and Python 3 different enough that perhaps Automake should 
have separate AM_PATH_PYTHON2 and AM_PATH_PYTHON3 macros, with 
AM_PATH_PYTHON deprecated, since few programs can usefully run with 
either Python 2 or 3?


In other words, are we conflating two features behind "PYTHON" that are 
actually logically separate?



-- Jacob

Re: PATCH: silence Shellcheck warning

[Jacob Bachmeyer]

Ben Elliston wrote:

On Tue, Jan 18, 2022 at 04:18:52AM -0500, Mike Frysinger wrote:

  

that said, automake targets lower than POSIX, so i agree that we
should leave it be for automake specifically.  feel like posting a
patch to suppress this particular check in lib/missing instead ?



If it's welcome, sure. Some people are opposed to littering pragmas
through source files to disable linters that they think do more harm
than good.


While I agree that Shellcheck's value is questionable, for the issues 
here there is no need to litter the file with pragmas.  Shellcheck will 
consider a pragma before the first executable line in the file to apply 
to the entire file; I used this when reverting $() to backticks in 
config.guess and have included the relevant excerpt below; 
the"shellcheck disable" line would similarly apply to missing but the 
rationale would need to be revised to explain the different issues.


8<--
diff --git a/config.guess b/config.guess
index d2bdee7..4bc8157 100755
--- a/config.guess
+++ b/config.guess
@@ -2,6 +2,8 @@
# Attempt to guess a canonical system name.
#   Copyright 1992-2021 Free Software Foundation, Inc.

+# shellcheck disable=SC2006,SC2268 # see below for rationale
+
timestamp='2021-05-24'

# This file is free software; you can redistribute it and/or modify it
@@ -32,6 +34,14 @@ timestamp='2021-05-24'
# Please send patches to .


+# The "shellcheck disable" line above the timestamp inhibits complaints
+# about features and limitations of the classic Bourne shell that were
+# superseded or lifted in POSIX.  However, this script identifies a wide
+# variety of pre-POSIX systems that do not have POSIX shells at all, and
+# even some reasonably current systems (Solaris 10 as case-in-point) still
+# have a pre-POSIX /bin/sh.
+
+
me=$(echo "$0" | sed -e 's,.*/,,')

usage="\
8<--



-- Jacob

Re: Automake for RISC-V

[Jacob Bachmeyer]

Billa Surendra wrote:

On Sun, 21 Nov, 2021, 2:28 am Nick Bowler,  wrote:
  

On 20/11/2021, Billa Surendra  wrote:


I have RISC-V native compiler on target image, but when I am compiling
automake on target image it needs automake on target. This is the main
problem.
  

Automake should not be required to install automake if you are using
a released version and have not modified the build system

Could you please explain more, What is the released version ? . Modified
build system means ?
  


Automake should only be needed if you have changed a "Makefile.am" file 
somewhere.


Are you using some kind of packaging system that likes to regenerate 
build files as a matter of course?  The normal "/path/to/src/configure 
&& make && make install" procedure should not require Automake to be 
installed.



-- Jacob

Re: Automake for RISC-V

[Jacob Bachmeyer]

Billa Surendra wrote:

Thanks for your reply. I have installed perl on target system but target
image and build system perl version were different. And second, thing I
have noticed  that in aclocal script very first line is #! /bin/perl
  


A simple workaround is to find perl on the target system image (probably 
/usr/bin/perl, but it could have been installed somewhere else) and make 
a symlink at /bin/perl to the real interpreter.  It is possible that 
your build system has /bin as a symlink to /usr/bin, as a certain 
widely-loathed developer has been rather forcefully advocating the past 
few years...




-- Jacob

[PATCH v1] Fix misuse of DejaGnu in Automake testsuite

[Jacob Bachmeyer]

This is a revised patch that changes the DejaGnu tests in the Automake 
testsuite to properly use a testsuite/ directory.  A previous "v0" of 
this patch was sent to the general Automake list as an illustration of 
the problem, but that version of the patch broke compatibility with all 
older releases of DejaGnu.  This version instead uses recursive make to 
run DejaGnu in the testsuite/ directory in each case.  This works with 
older versions of DejaGnu and this usage of DejaGnu will continue to be 
supported for the foreseeable future.


On a side note, DejaGnu can now be run from its source tree or a Git 
working tree.  While the Automake testsuite does not seem to provide a 
way to set RUNTEST to point to an alternate runtest, " 
PATH=/where/you/put/dejagnu-1.6.3:$PATH make check " will use an 
unpacked version of the 1.6.3 release even if the system has an older 
version installed.  Older versions of DejaGnu did not support this reliably.


8<--
diff -urN -x '*~' automake-1.16.3/t/check12.sh 
automake-1.16.3-patch/t/check12.sh
--- automake-1.16.3/t/check12.sh2020-11-18 19:21:03.0 -0600
+++ automake-1.16.3-patch/t/check12.sh  2021-07-13 21:49:22.900870882 -0500
@@ -22,6 +22,7 @@
. test-init.sh

cat >> configure.ac << 'END'
+AC_CONFIG_FILES([testsuite/Makefile])
AC_OUTPUT
END

@@ -56,12 +57,17 @@
## DejaGnu tests.

cat >> Makefile.am << 'END'
-AUTOMAKE_OPTIONS += dejagnu
+SUBDIRS = testsuite
+EXTRA_DIST += hammer spanner
+END
+
+mkdir testsuite
+
+cat >> testsuite/Makefile.am << 'END'
+AUTOMAKE_OPTIONS = dejagnu
DEJATOOL = hammer spanner
-AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer SPANNER=$(srcdir)/spanner
-EXTRA_DIST += $(DEJATOOL)
-EXTRA_DIST += hammer.test/hammer.exp
-EXTRA_DIST += spanner.test/spanner.exp
+AM_RUNTESTFLAGS = HAMMER=$(top_srcdir)/hammer SPANNER=$(top_srcdir)/spanner
+EXTRA_DIST = hammer.test/hammer.exp spanner.test/spanner.exp
END

cat > hammer << 'END'
@@ -77,9 +83,9 @@
END
chmod +x hammer spanner

-mkdir hammer.test spanner.test
+mkdir testsuite/hammer.test testsuite/spanner.test

-cat > hammer.test/hammer.exp << 'END'
+cat > testsuite/hammer.test/hammer.exp << 'END'
set test test_hammer
spawn $HAMMER
expect {
@@ -88,7 +94,7 @@
}
END

-cat > spanner.test/spanner.exp << 'END'
+cat > testsuite/spanner.test/spanner.exp << 'END'
set test test_spanner
spawn $SPANNER
expect {
@@ -133,10 +139,10 @@
  $srcdir/configure

  $MAKE check
-  test -f hammer.log
-  test -f hammer.sum
-  test -f spanner.log
-  test -f spanner.sum
+  test -f testsuite/hammer.log
+  test -f testsuite/hammer.sum
+  test -f testsuite/spanner.log
+  test -f testsuite/spanner.sum
  if test x"$am_serial_tests" != x"yes"; then
test -f test-suite.log
test -f a.log
@@ -147,10 +153,10 @@
  cp -f config.status config-status.sav

  $MAKE distclean
-  test ! -e hammer.log
-  test ! -e hammer.sum
-  test ! -e spanner.log
-  test ! -e spanner.sum
+  test ! -e testsuite/hammer.log
+  test ! -e testsuite/hammer.sum
+  test ! -e testsuite/spanner.log
+  test ! -e testsuite/spanner.sum
  test ! -e test-suite.log
  test ! -e a.log
  test ! -e b.log
@@ -160,12 +166,12 @@
  ./config.status

  NAIL=screw $MAKE check && exit 1
-  test -f hammer.log
-  test -f hammer.sum
-  test -f spanner.log
-  test -f spanner.sum
-  grep 'FAIL: test_hammer' hammer.sum
-  grep 'FAIL:' spanner.sum && exit 1
+  test -f testsuite/hammer.log
+  test -f testsuite/hammer.sum
+  test -f testsuite/spanner.log
+  test -f testsuite/spanner.sum
+  grep 'FAIL: test_hammer' testsuite/hammer.sum
+  grep 'FAIL:' testsuite/spanner.sum && exit 1

  B_EXIT_STATUS=1 $MAKE check && exit 1
  if test x"$am_serial_tests" != x"yes"; then
@@ -185,12 +191,12 @@

  # Do not trust the exit status of 'make -k'.
  NAIL=screw B_EXIT_STATUS=23 CHECKLOCAL_EXIT_STATUS=1 $MAKE -k check || :
-  test -f hammer.log
-  test -f hammer.sum
-  test -f spanner.log
-  test -f spanner.sum
-  grep 'FAIL: test_hammer' hammer.sum
-  grep 'FAIL:' spanner.sum && exit 1
+  test -f testsuite/hammer.log
+  test -f testsuite/hammer.sum
+  test -f testsuite/spanner.log
+  test -f testsuite/spanner.sum
+  grep 'FAIL: test_hammer' testsuite/hammer.sum
+  grep 'FAIL:' testsuite/spanner.sum && exit 1
  if test x"$am_serial_tests" != x"yes"; then
cat test-suite.log
cat a.log
diff -urN -x '*~' automake-1.16.3/t/dejagnu3.sh 
automake-1.16.3-patch/t/dejagnu3.sh
--- automake-1.16.3/t/dejagnu3.sh   2020-11-18 19:21:03.0 -0600
+++ automake-1.16.3-patch/t/dejagnu3.sh 2021-07-13 22:04:45.653614878 -0500
@@ -27,19 +27,27 @@
chmod +x hammer

cat >> configure.ac << 'END'
+AC_CONFIG_FILES([testsuite/Makefile])
AC_OUTPUT
END

cat > Makefile.am << 'END'
+SUBDIRS = testsuite
+EXTRA_DIST = hammer
+END
+
+mkdir testsuite
+
+cat > testsuite/Makefile.am << 'END'
AUTOMAKE_OPTIONS = dejagnu
DEJATOOL = hammer
-AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer
-EXTRA_DIST = hammer hammer.test/hammer.exp
+AM_RUNTESTFLAGS = HAMMER=$(top_srcdir)/hammer
+EXTRA_DIST = hammer.test/hammer.exp
END

-mkdir 

Re: Automake testsuite misuses DejaGnu [PATCH v0]

[Jacob Bachmeyer]

Jim Meyering wrote:

[...]
Even a sample fix for one of the currently-failing tests would be helpful.
  


This is the first draft; this patch breaks 1.6.1 because versions of 
DejaGnu prior to 1.6.3 require srcdir to point exactly to the testsuite, 
while 1.6.3 allows the testsuite to be in ${srcdir}/testsuite.


8<--
diff -urN -x '*~' automake-1.16.3-original/t/check12.sh 
automake-1.16.3/t/check12.sh
--- automake-1.16.3-original/t/check12.sh   2020-11-18 19:21:03.0 
-0600
+++ automake-1.16.3/t/check12.sh2021-06-29 01:47:21.669276386 -0500
@@ -60,8 +60,8 @@
DEJATOOL = hammer spanner
AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer SPANNER=$(srcdir)/spanner
EXTRA_DIST += $(DEJATOOL)
-EXTRA_DIST += hammer.test/hammer.exp
-EXTRA_DIST += spanner.test/spanner.exp
+EXTRA_DIST += testsuite/hammer.test/hammer.exp
+EXTRA_DIST += testsuite/spanner.test/spanner.exp
END

cat > hammer << 'END'
@@ -77,9 +77,10 @@
END
chmod +x hammer spanner

-mkdir hammer.test spanner.test
+mkdir testsuite
+mkdir testsuite/hammer.test testsuite/spanner.test

-cat > hammer.test/hammer.exp << 'END'
+cat > testsuite/hammer.test/hammer.exp << 'END'
set test test_hammer
spawn $HAMMER
expect {
@@ -88,7 +89,7 @@
}
END

-cat > spanner.test/spanner.exp << 'END'
+cat > testsuite/spanner.test/spanner.exp << 'END'
set test test_spanner
spawn $SPANNER
expect {
diff -urN -x '*~' automake-1.16.3-original/t/dejagnu3.sh 
automake-1.16.3/t/dejagnu3.sh
--- automake-1.16.3-original/t/dejagnu3.sh  2020-11-18 19:21:03.0 
-0600
+++ automake-1.16.3/t/dejagnu3.sh   2021-06-29 01:19:19.161147525 -0500
@@ -34,12 +34,13 @@
AUTOMAKE_OPTIONS = dejagnu
DEJATOOL = hammer
AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer
-EXTRA_DIST = hammer hammer.test/hammer.exp
+EXTRA_DIST = hammer testsuite/hammer.test/hammer.exp
END

-mkdir hammer.test
+mkdir testsuite
+mkdir testsuite/hammer.test

-cat > hammer.test/hammer.exp << 'END'
+cat > testsuite/hammer.test/hammer.exp << 'END'
set test test
spawn $HAMMER
expect {
diff -urN -x '*~' automake-1.16.3-original/t/dejagnu4.sh 
automake-1.16.3/t/dejagnu4.sh
--- automake-1.16.3-original/t/dejagnu4.sh  2020-11-18 19:21:03.0 
-0600
+++ automake-1.16.3/t/dejagnu4.sh   2021-06-29 01:25:08.309780437 -0500
@@ -49,13 +49,14 @@

AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer SPANNER=$(srcdir)/spanner

-EXTRA_DIST  = hammer  hammer.test/hammer.exp
-EXTRA_DIST += spanner spanner.test/spanner.exp
+EXTRA_DIST  = hammer  testsuite/hammer.test/hammer.exp
+EXTRA_DIST += spanner testsuite/spanner.test/spanner.exp
END

-mkdir hammer.test spanner.test
+mkdir testsuite
+mkdir testsuite/hammer.test testsuite/spanner.test

-cat > hammer.test/hammer.exp << 'END'
+cat > testsuite/hammer.test/hammer.exp << 'END'
set test test
spawn $HAMMER
expect {
@@ -64,7 +65,7 @@
}
END

-cat > spanner.test/spanner.exp << 'END'
+cat > testsuite/spanner.test/spanner.exp << 'END'
set test test
spawn $SPANNER
expect {
diff -urN -x '*~' automake-1.16.3-original/t/dejagnu5.sh 
automake-1.16.3/t/dejagnu5.sh
--- automake-1.16.3-original/t/dejagnu5.sh  2020-11-18 19:21:03.0 
-0600
+++ automake-1.16.3/t/dejagnu5.sh   2021-06-29 01:26:36.511645792 -0500
@@ -34,12 +34,13 @@

cat > Makefile.am << END
AUTOMAKE_OPTIONS = dejagnu
-EXTRA_DIST = $package $package.test/$package.exp
+EXTRA_DIST = $package testsuite/$package.test/$package.exp
AM_RUNTESTFLAGS = PACKAGE=\$(srcdir)/$package
END

-mkdir $package.test
-cat > $package.test/$package.exp << 'END'
+mkdir testsuite
+mkdir testsuite/$package.test
+cat > testsuite/$package.test/$package.exp << 'END'
set test "a_dejagnu_test"
spawn $PACKAGE
expect {
diff -urN -x '*~' automake-1.16.3-original/t/dejagnu6.sh 
automake-1.16.3/t/dejagnu6.sh
--- automake-1.16.3-original/t/dejagnu6.sh  2020-11-18 19:21:03.0 
-0600
+++ automake-1.16.3/t/dejagnu6.sh   2021-06-29 01:28:07.151396859 -0500
@@ -35,8 +35,9 @@
AM_RUNTESTFLAGS = FAILDEJA=$(srcdir)/faildeja
END

-mkdir faildeja.test
-cat > faildeja.test/faildeja.exp << 'END'
+mkdir testsuite
+mkdir testsuite/faildeja.test
+cat > testsuite/faildeja.test/faildeja.exp << 'END'
set test failing_deja_test
spawn $FAILDEJA
expect {
diff -urN -x '*~' automake-1.16.3-original/t/dejagnu7.sh 
automake-1.16.3/t/dejagnu7.sh
--- automake-1.16.3-original/t/dejagnu7.sh  2020-11-18 19:21:03.0 
-0600
+++ automake-1.16.3/t/dejagnu7.sh   2021-06-29 01:29:38.877097021 -0500
@@ -39,8 +39,9 @@
AM_RUNTESTFLAGS = --status FAILTCL=$(srcdir)/failtcl
END

-mkdir failtcl.test
-cat > failtcl.test/failtcl.exp << 'END'
+mkdir testsuite
+mkdir testsuite/failtcl.test
+cat > testsuite/failtcl.test/failtcl.exp << 'END'
set test test
spawn $FAILTCL
expect {
diff -urN -x '*~' automake-1.16.3-original/t/dejagnu-absolute-builddir.sh 
automake-1.16.3/t/dejagnu-absolute-builddir.sh
--- automake-1.16.3-original/t/dejagnu-absolute-builddir.sh 2020-11-18 
19:21:03.0 -0600
+++ automake-1.16.3/t/dejagnu-absolute-builddir.sh  2021-06-29 

Re: Automake testsuite misuses DejaGnu

[Jacob Bachmeyer]

Daniel Herring wrote:
It seems fragile for DejaGnu to probe for a testsuite directory and 
change its behavior as you describe.  For example, I could have a 
project without the testsuite dir, invoke the tester, and have it find 
and run some unrelated files in the parent directory.  Unexpected 
behavior (chaos) may ensue.


This already happens and this is the behavior that is deprecated and 
even more fragile.  Without a testsuite/ directory, DejaGnu will end up 
searching the tree for *.exp files and running them all.  Eventually, if 
$srcdir neither is nor contains "testsuite", DejaGnu will throw an error 
and abort.  The testsuite/ directory is a long-documented requirement.


Is there an explicit command-line argument that could be added to the 
Automake invocation?


Not easily; the probing is done specifically to allow for two different 
ways of using DejaGnu:  using recursive Makefiles that invoke DejaGnu 
with the testsuite/ directory current, and using non-recursive 
Makefiles, which with Automake will invoke DejaGnu with the top-level 
directory, presumably containing the "testsuite" directory.  Both of 
these cases must be supported:  the toolchain packages use the former 
and Automake's basic DejaGnu support will use the latter if a 
non-recursive layout is desired.


Both of these use the same command line argument --srcdir and site.exp 
variable srcdir; the difference is that srcdir has acquired two 
different meanings.



-- Jacob

Re: Automake testsuite misuses DejaGnu

[Jacob Bachmeyer]

Karl Berry wrote:
DejaGnu has always required a DejaGnu testsuite to be rooted at a 
"testsuite" directory


If something was stated in the documentation, but not enforced by the
code, hardly surprising that "non-conformance" is widespread.
  


It is not widespread -- all of the toolchain packages correctly place 
their testsuites in testsuite/ directories.  As far as I know, the 
Automake tests are the only outlier.



Anyway, it seems like an unfriendly requirement for users. And even more
to incompatibly enforce something now that has not been enforced for
previous decades. Why? (Just wondering.) -k


Previous versions of DejaGnu did not properly handle non-recursive make 
with Automake-produced makefiles.  Beginning with 1.6.3, the testsuite 
is allowed to be in ${srcdir}/testsuite instead of ${srcdir} exactly.  
Enforcing the long-documented (and mostly followed) requirement that 
there be a directory named "testsuite" containing the testsuite allows 
DejaGnu to resolve the ambiguity and determine if it has been invoked at 
package top-level or in the testsuite/ directory directly.


Even in 1.6.3, there was intent to continue to allow the broken cases to 
work with a warning, but I made the conditional for that case too narrow 
(oops!) and some of the Automake test cases fail as a result.  Fixing 
this now is appropriate because no one is going to see the future 
deprecation warnings due to the way Automake tests are run.



-- Jacob

Re: Automake testsuite misuses DejaGnu

[Jacob Bachmeyer]

Jim Meyering wrote:

On Sun, Jul 11, 2021 at 9:03 PM Jacob Bachmeyer  wrote:
  

[...]

The affected tests are:  check12, dejagnu3, dejagnu4, dejagnu5,
dejagnu6, dejagnu7, dejagnu-absolute-builddir, dejagnu-relative-srcdir,
dejgnu-siteexp-extend, dejagnu-siteexp-useredit.

[...]



Thank you for the analysis and heads-up.
I see that Fedora 34 currently has only dejagnu-1.6.1.
If this is something you can help with now, I can certainly wait a few days.

Even a sample fix for one of the currently-failing tests would be helpful.
  


That is part of the problem:  I have a patch, but applying it will cause 
the tests to fail with DejaGnu 1.6.1.  Older versions of DejaGnu require 
$srcdir to be exactly the root of the testsuite, while 1.6.3 accepts a 
testsuite in $srcdir or ${srcdir}/testsuite; the latter is needed to 
allow Automake to invoke DejaGnu from the top-level in the tree.


I expect to have time to try a recursive make solution later tonight or 
tomorrow.  Do I understand correctly that I will need to add "SUBDIRS = 
testsuite" to the top-level TEST_CASE/Makefile.am in the test case and 
move the "AUTOMAKE_OPTIONS = dejagnu" and "DEJATOOL" definitions to 
TEST_CASE/testsuite/Makefile.am to get Automake to invoke DejaGnu in the 
testsuite subdirectory instead of top-level?



-- Jacob

Automake testsuite misuses DejaGnu

[Jacob Bachmeyer]

I was planning to find a solution with a complete patch before 
mentioning this, but since a release is imminent I will just state the 
problem:  several tests in the Automake testsuite misuse DejaGnu and 
fail with the 1.6.3 DejaGnu release as a result.


DejaGnu has always required a DejaGnu testsuite to be rooted at a 
"testsuite" directory and this has long been documented in the manual.  
However, prior to 1.6.3, DejaGnu did not actually depend on this 
requirement being met.  Changes during the development process to 
properly support non-recursive Automake makefiles required relying on 
this requirement to resolve the ambiguity between recursive and 
non-recursive usage.  Several tests in the Automake testsuite do not 
meet this requirement and fail if run with DejaGnu 1.6.3.


The simple change of updating the tests to use a testsuite/ directory 
causes the tests to fail with older versions of DejaGnu, due to lack of 
support for non-recursive "make check" in those versions.  I have not 
yet tried a patch that also switches the tests to use recursive make, 
but I believe that is probably the only way for the tests to pass with 
old and new DejaGnu.


Note that, according to the original author, Rob Savoye, DejaGnu has 
always been intended to require that testsuites be rooted at a 
"testsuite" directory and the behavior that Automake's test cases rely 
on was never supported.


The affected tests are:  check12, dejagnu3, dejagnu4, dejagnu5, 
dejagnu6, dejagnu7, dejagnu-absolute-builddir, dejagnu-relative-srcdir, 
dejgnu-siteexp-extend, dejagnu-siteexp-useredit.


Note that these tests do not all fail with the 1.6.3 release, but will 
all fail with some future release when the undocumented support for a 
testsuite not rooted at "testsuite" will eventually be removed.



-- Jacob

Re: parallel build issues

[Jacob Bachmeyer]

Bob Friesenhahn wrote:
It is possible to insert additional dependency lines in Makefile.am so 
software is always built in the desired order, but this approach might 
only work if you always build using the top level Makefile.


This should actually work here:  the problem is that a target in doc/ 
also depends on a target in frontend/ and uses recursive make to build 
that target.  When the top-level Makefile is used in parallel mode, 
sub-makes are concurrently run in both doc/ and frontend/ but the doc/ 
sub-make invokes another make in frontend/ leading to a race and failure.


If only doc/Makefile is used, it will spawn a sub-make in frontend/ that 
will be the only make running there and will succeed.  If only 
frontend/Makefile is used, everything works similarly.  Since the 
problem can only occur when building with the top-level Makefile, adding 
a dependency in the top-level Makefile should prevent it.



-- Jacob

Re: Add support for EXTRA_DIST_LINKS

[Jacob Bachmeyer]

madmurphy wrote:

But what if a developer wants to package something *for another developer*? It 
is not unusual to have a non-bootstraped “dev” version of a package circulating 
among developers


How is Git not adequate for this?  If you need to pass files around, 
there is the "git bundle" command.



-- Jacob

Re: Add support for EXTRA_DIST_LINKS

[Jacob Bachmeyer]

madmurphy wrote:

P.S. *“If I remember correctly, Autoconf also falls back to hardlinks or
copying the files if the system does not support symbolic links.”*: Same
goes for my patch. I use $(LN_S) for creating the links, and that falls
back to hardlinks when ln -s is not supported.


The difference is that your patch will fall back if the /development/ 
system does not support symlinks, while AC_CONFIG_LINKS will fall back 
if the /build/ system does not support symlinks.


Typically, the development system /will/ support symlinks, so symlinks 
will be recorded in the distribution tarball.  The GNU coding standards 
are vehemently against this because such a tarball looks fine to the 
developer, but cannot even be properly unpacked on a system that does 
not support symlinks.


What can EXTRA_DIST_LINKS achieve that AC_CONFIG_LINKS cannot?


-- Jacob

Re: Add support for EXTRA_DIST_LINKS

[Jacob Bachmeyer]

madmurphy wrote:

I have created a patch for Automake to support a new Makefile variable
named EXTRA_DIST_LINKS. The purpose of the variable is that of creating
symlinks for the distributed archive.

The syntax is very simple: a space-separated list of tokens of the form
DEST:SOURCE – it is identical to that of AC_CONFIG_LINKS(dest:source ...)
in the configure.ac file.
  


The GNU coding standards specifically say not to do this:
"Don’t include any symbolic links in the distribution itself. If the tar 
file contains symbolic links, then people cannot even unpack it on 
systems that don’t support symbolic links. Also, don’t use multiple 
names for one file in different directories, because certain file 
systems cannot handle this and that prevents unpacking the 
distribution." (section 7.3 "Making Releases")


AC_CONFIG_LINKS makes the links when configure is run, after the 
distribution has been unpacked.  If I remember correctly, Autoconf also 
falls back to hardlinks or copying the files if the system does not 
support symbolic links.



-- Jacob