Re: m4_map and AC_REQUIRE hoisting

[Nick Bowler]

Hi Eric,

On 1/24/18, Eric Blake  wrote:
> On 01/24/2018 12:32 AM, Nick Bowler wrote:
>> Hello Autoconfers,
>>
>> I hit a weird (to me) issue involving m4_map(all) and AC_REQUIRE, and I
>> would appreciate some help understanding what's going on here!
>
> Let's see if I can give an explanation.

Thanks for the explanation.  Indeed, when thinking about this more (and
not at 01:00) I recalled these two tidbits from the autoconf manual:

  - "some of the topological sorting algorithms used in resolving macro
dependencies use diversions", and

  - "it is important to be aware of an M4 limitation regarding
diversions: text only goes to a diversion if it is not part of
argument collection"

which sure enough suggests an underquoting issue that you point out,
and removing some quotes from my "working" examples produces the same
"unexpected" output.

[snip TEST0 bits]
>> But when my macro is using m4_map or m4_mapall, this mechanism seems to
>> break down.  Example:
>>
>>   AC_INIT([test], [0])
>>
>>   AC_DEFUN([DEF1], [echo def1])
>>   AC_DEFUN([REQ1], [AC_REQUIRE([DEF1])])
>>
>>   AC_DEFUN([TEST1], [m4_mapall([m4_newline], [[echo hello], [REQ1]])])
>
> Underquoted; the second argument to m4_mapall() is a "comma-separated
> quoted list of argument descriptions" which are "in turn a
> comma-separated quoted list of quoted elements".  Better would be:
>
> m4_mapall([m4_newline], [[[echo hello]], [[REQ1]]])

I see.  Indeed this is all spelled out clearly in the manual and despite
rechecking this many times I failed to actually understand what m4_map
is expecting for its 2nd argument.

I think the fact that m4_map could pass anything other than one argument
to the "mapped" macro did not even enter my head as a possibility, and
the examples presented even demonstrate this usage!

[snip helpful explanation]
[...]
>>   m4_define([my_mapall], [m4_ifval([$2],
>> [m4_indir([$1], m4_car($2))[]my_mapall([$1], m4_cdr($2))])])
>>   AC_DEFUN([TEST2], [my_mapall([m4_newline], [[echo hello], [REQ2]])])
>
> Your naive version is less susceptible to underquoting issues [...]

I think the important thing here is that my version always passes
exactly one argument to the "mapped" macro, so it isn't the same
operation as m4_mapall and explains the key difference in results!

That is, it reflects what I /imagined/ m4_mapall does rather than what
it /actually/ does... :)

> Hopefully my explanation into the deep magic guts helped!

Perfectly.

Thanks,
  Nick

___
Autoconf mailing list
Autoconf@gnu.org
https://lists.gnu.org/mailman/listinfo/autoconf

Re: m4_map and AC_REQUIRE hoisting

[Eric Blake]

On 01/24/2018 12:32 AM, Nick Bowler wrote:
> Hello Autoconfers,
> 
> I hit a weird (to me) issue involving m4_map(all) and AC_REQUIRE, and I
> would appreciate some help understanding what's going on here!

Let's see if I can give an explanation.

> 
> So normally when expanding a macro defined with AC_DEFUN and a nested
> AC_REQUIRE causes a macro to be expanded, that expansion gets "hoisted"
> outside of the topmost macro expansion.  Example:
> 
>   AC_INIT([test], [0])
>   
>   AC_DEFUN([DEF0], [echo def0])
>   AC_DEFUN([REQ0], [AC_REQUIRE([DEF0])])
>   
>   AC_DEFUN([TEST0], [m4_newline([echo hello])[]m4_newline([REQ0])])
>   TEST0
> 
> When I run this, I see:
> 
>   def0
>   hello
> 
> which is as expected.

Or, put in other words, expanding an AC_REQUIRE tells autoconf to "make
sure the required macro gets expanded prior to the start of the macro
I'm currently expanding".  Step-wise, you are expanding:

TEST0

into

m4_newline([echo hello])[]m4_newline([REQ0])

into:


echo hello
REQ0

into:


echo hello
AC_REQUIRE([DEF0])

where AC_REQUIRE figures out the current context (that of outputting
TEST0), and ensures that DEF0 is expanded prior to that point, thus
behaving like:

DEF0

echo hello

and turning into the desired:

echo def0
echo hello

> 
> But when my macro is using m4_map or m4_mapall, this mechanism seems to
> break down.  Example:
> 
>   AC_INIT([test], [0])
> 
>   AC_DEFUN([DEF1], [echo def1])
>   AC_DEFUN([REQ1], [AC_REQUIRE([DEF1])])
> 
>   AC_DEFUN([TEST1], [m4_mapall([m4_newline], [[echo hello], [REQ1]])])

Underquoted; the second argument to m4_mapall() is a "comma-separated
quoted list of argument descriptions" which are "in turn a
comma-separated quoted list of quoted elements".  Better would be:

m4_mapall([m4_newline], [[[echo hello]], [[REQ1]]])

(there are two argument descriptions, [[echo hello]] and [[REQ1]]; the
first argument description is the quoted list containing the single
element [echo hello] as the argument to supply).

>   TEST1
> 
> When I run that, I get:
> 
>   hello
>   def1
> 
> with the order reversed from my expectation...

step-wise, this is going from

TEST1

into

m4_mapall([m4_newline], [[echo hello], [REQ1]])

and conceptually into this (because of the underquoting):

m4_newline(echo hello)[]m4_newline(REQ1)[]

But note what happens here.  Since REQ1 is unquoted, it gets expanded
PRIOR to invoking m4_newline(), in order to determine what argument to
pass to m4_newline; and as currently implemented, the require mechanism
in autoconf can only hoist its argument to the front of the current
parsing level.  So you are getting:


echo hello[]m4_newline(AC_REQUIRE([DEF1]))[]

then:


echo hello[]m4_newline(DEF1)[]


echo hello[]m4_newline(echo def1)[]

where the required text occurs before anything else at the current
parsing level (but the current parsing level was during argument
collection), so the requirement text is passed as the argument to
m4_newline(), at which point you end with:


echo hello
echo def1

and the order is different than you wanted, because you did not
encounter AC_REQUIRE during the context of the body of TEST1 for
hoisting anything prior to TEST1.


And indeed, when I retried your example with sufficient quoting, I got
the desired 'def1' before 'hello', because the expansion of REQ1 is then
deferred until after m4_newline() has done its work, at which point the
parse is now in the context of expanding TEST1 rather than collecting
arguments to m4_newline.

AC_DEFUN([TEST1], [m4_mapall([m4_newline], [[[echo hello]], [[REQ1]]])])


> 
> But if I implement my own version of mapall in a straightforward way,
> it seems to work out just fine:
> 
>   AC_INIT([test], [0])
>   
>   AC_DEFUN([DEF2], [echo def2])
>   AC_DEFUN([REQ2], [AC_REQUIRE([DEF2])])
>   
>   m4_define([my_mapall], [m4_ifval([$2],
> [m4_indir([$1], m4_car($2))[]my_mapall([$1], m4_cdr($2))])])
>   AC_DEFUN([TEST2], [my_mapall([m4_newline], [[echo hello], [REQ2]])])

Your naive version is less susceptible to underquoting issues, but is
more expensive in computation power (it computes the expansion of $2
much more frequently, and therefore m4 has to expand more input - adding
a quadratic complexity for how many bytes are parsed as the list gets
larger) - I remember spending quite a bit of time optimizing m4_map and
friends to get to a minimal expansion that autoconf currently uses; for
small lists (like your example of two argument descriptions with one
argument each), the difference is in the noise, but for large lists
(such as hundreds of argument descriptions, where I seem to recall that
AC_CHECK_FUNCS_ONCE was a candidate for something that could map lots of
arguments in a configure.ac that checks a lot of functions), the
difference between O(n^3) and O(n^2) algorithms was very noticeable in
time and memory consumed by 'autoconf'.

>   TEST2
> 
> Running this gives:
> 
>   def2
>   hello
> 
> as expected.
> 
> I really have no idea why TEST1 results in a different