The following issue has been SUBMITTED.
======================================================================
http://austingroupbugs.net/view.php?id=1105
======================================================================
Reported By: stephane
Assigned To:
======================================================================
Project: 1003.1(2016)/Issue7+TC2
Issue ID: 1105
Category: Shell and Utilities
Type: Enhancement Request
Severity: Editorial
Priority: normal
Status: New
Name: Stéphane Chazelas
Organization:
User Reference:
Section: awk
Page Number:
Line Number:
Interp Status: ---
Final Accepted Text:
======================================================================
Date Submitted: 2016-12-05 21:52 UTC
Last Modified: 2016-12-05 21:52 UTC
======================================================================
Summary: problems with backslashes in awk strings and EREs
Description:
In awk's string constants ("string"), \ is used to quote itself and "
but also to introduce escape sequences like \n, \b, \56...
That is specified in the spec.
The spec does also specify the fact that that also happens for
awk -v var='string' and awk '{code}' var='string', though I wish
it made it clearer and for instance added a warning along the
lines of "applications wishing to pass arbitrary strings by way
of -v var="$value" or var="$value", should make sure backslashes
are escaped or use the ENVIRON or ARGV methods instead" as that's
a source of many bugs.
Now, the spec gives the same escaping rules for "...", /ERE/, -v
var=... and var=...
In particular, it says \/ means a slash for all and not only for
/ERE/ or that \" means a " for all and not only for "...".
IOW, it says -v a='\@' for instance is unspecified, but that
-v a='\"' is specified and means the same as -v a='"'
In practice that's not the case.
Solaris 10:
$ /usr/xpg4/bin/awk -v 'a=\"' 'BEGIN {print a}'
\"
$ gawk 'BEGIN{print "\/"}'
gawk: cmd. line:1: warning: escape sequence `\/' treated as plain `/'
/
When it comes to regexps and escape sequences, it gets worse.
It looks like the spec specifies (to some extent) some accident
of implementation of the historical awk implementation.
It requires for instance that both:
"\f" ~ "\f"
and
"\f" ~ "\\f"
be true. The first one matching against an ERE that consists in
a formfeed character. The second again an "\f" ERE that is meant
to match the formfeed character.
IOW, the backslash-f sequence is understood as a RE operator.
And /ERE/ is not a RE quoting operator that allows for escape
sequences like "...", but the /.../ are like strong quotes and
it's up to the regexp engine to interpret those \f.
If I understand correctly, it requires "x" ~ /\56/ (on ASCII
systems where \56 is ".") to return false.
While that's what I observe with bwk's awk (or FreeBSD awk based
on it), that's not what I observe with many other modern (even
certified) awk implementations:
$ /usr/xpg4/bin/awk 'BEGIN{print "\f" ~ "\\f"}'
0
$ /usr/xpg4/bin/awk 'BEGIN{print "x" ~ /\56/}'
1
$ gawk 'BEGIN{print "\f" ~ "\\f"}'
1
$ gawk 'BEGIN{print "x" ~ /\56/}'
1
$ busybox awk 'BEGIN{print "\f" ~ "\\f"}'
0
$ busybox awk 'BEGIN{print "x" ~ /\56/}'
1
Also, with the confusion between "\" as a RE quoting operator,
as the " and / escaping operator and its role in ANSI C escape
sequence expansion, it is unclear how we should understand \
inside bracket expressions.
It says awk EREs should be the POSIX EREs (where \ is not
special within [...]), and also that the ansi escape sequences
should be supported inside bracket expressions. \\ is listed as
a C escape expansion. Anything other than the list given is
unspecified, so it's even unclear whether /\./ is specified or
if /\\./ means the \. or \\. ERE (if it were not for the example
section that suggests the lattern). In ERE, [\xy] matches \, x
or y, what should it be in awk?
Note that I can find awk implementations (like nawk on Solaris)
where "[]]" or "[^]]" can't be used to match a "]" or non-].
Most awk implementations I've tried work with [\]] or [^\]], the
only exception I've found being Solaris 10 awk.
Also it doesn't make it clear what "\777" should expand to.
Historical implementations expand to the same as "\377" but some
expand to the same as "\0777" (taken as \77 followed by 7).
Desired Action:
It should be possible to have the spec cover most of the
existing behaviour without affecting functionality too much as
implementations currently are non-compliant mostly in areas that
are corner cases unlikely to be used in practice. IOW, I think
the spec should stop requiring behaviours that could be regarded
as accidents of implementation of historic nawk
implementations.
When it comes to \", \/, they should only be specified to escape
the corresponding delimiter for /ERE/ and "string".
"\/" -> unspecified
-v a='\/' -> unspecified
-v a='\"' -> unspecified
/\"/ -> unspecified
/\// -> specified
"\"" -> specified
Don't make the \a, \b... part of the ERE syntax. awk EREs to be
POSIX EREs except that within bracket expressions "\" has to be
doubled ($0 ~ "[\\x]" (or /[\x]/) unspecified, has to be
$0 ~ "[\\\\x]" (or /[\\x]/) to match \ or x)
In the ERE syntax, \a, \b... to be unspecified like in normal EREs
($0 ~ "\\a" unspecified), but the /ERE/ syntax allows for them
(/\a/ or /[\a]/ shall match the BEL character).
In /ERE/ make it unspecified whether those expansions happen
before or as part of the regex matching (unspecified whether /\56/
on ASCII-based systems matches any character or just "." or if
[a\55z] is like [a-z] or [-az]).
However make it clear that outside of bracket expressions /\X/
where X is an ERE operator (including \ itself), means the \X
ERE. (matches the X character literally).
Since POSIX only supports 8-bit bytes, and to allow
implementations to treat \456 as \0456, behaviour unspecified
for \ddd where ddd > 377.
Please also consider adding a note about -v var=value or
assignment arguments having problems with backslashes and the
safer ENVIRON/ARGV alternatives.
Features we'd lose with that change:
PATTERN='\f' awk '$0 ~ ENVIRON["PATTERN"]'
currently required to match a formfeed would cease to be
specified. But in practice, that already fails with some
implementations like Solaris /usr/xpg4/bin/awk or busybox awk.
LC_ALL=C awk '$0 ~ /[\123-\234]/'
could not be used reliably for some values in place of 123 or
234 (like the code point for "^" or "]" instead). Again, in
practice that's already the case with several implementations.
======================================================================
Issue History
Date Modified Username Field Change
======================================================================
2016-12-05 21:52 stephane New Issue
2016-12-05 21:52 stephane Name => Stéphane Chazelas
2016-12-05 21:52 stephane Section => awk
======================================================================
