2018-05-04 16:10:25 +0100, Geoff Clare:
[...]
> It was a deliberate choice made by the original POSIX.2 developers.
> See XRAT A.9.3.5:
>
> Current practice in awk and lex is to accept escape sequences in
> bracket expressions as per XBD Table 5-1 (on page 121), while the
> normal ERE behavior is to regard such a sequence as consisting of
> two characters. Allowing the awk/lex behavior in EREs would change
> the normal behavior in an unacceptable way; it is expected that
> awk and lex will decode escape sequences in EREs before passing
> them to regcomp() or comparable routines.
[...]
I'd say it was an unfortunate decision. \ treated specially
inside bracket expressions in practice is more common than the
behaviour POSIX specifies where it's not treated specially.
I'm fine that POSIX requires []xyz] to match on ] and [xy-] on -
and that it's the only portable way to match a ] or - inside a
bracket expression. I'd much rather use [xy\]z] or [x\-y]
instead of having to make sure ] is first and - is first or
last, but I understand some implementations don't support it
(note the complication with wildcards where you can't do [^!]
nor [!^] to match on either ^ or !).
I'm not fine with POSIX requiring [\t] to match on backslash and
t, [\-] to match on backslash and -. Telling application writers
that they can portably use [\-] to match either \ or - is a lie.
In practice in many tools, you need [\\-] if not [-\\].
By requiring that, POSIX shot them in the foot. They shut the
door to possible extensions.
With PCRE or compatible now the defacto standard for regexp
about anywhere else (perl, python, ruby, php, js...), people
expect \d to match decimal digits, \s on spacing characters, \t
to match tab... both inside and outside bracket expressions.
Things like sed/grep/ex/vi can't do it inside bracket expressions
if they want to be POSIX compliant. GNU sed and vim do it for
\t, \b (not \s, \d) in a conformance mode (POSIXLY_CORRECT for
GNU sed, "compatible"/"cp" option in vim). vim still doesn't match
"\" with [\-] after set cp.
(sure , POSIX still allows extensions like [[:tab:]] instead of
[\t] but I don't see any implementation doing that. I myself use
\s when I can avoid using [[:space:]] on the command line even
if it means firing perl instead of sed)
How would a ticket that asks that the bracket expression
specification be modified to make clear that applications need to
double the "\" for "\" to be matched be recieved?
[...]
> > Does the change address the differences in behaviour for
> >
> > PATTERN='\f' awk '$0 ~ $ENVIRON["PATTERN"]'
> > or
> > awk '$0 ~ "\\f"'
>
> This was discussed in a conference call and we decided no change
> was needed as the standard already says the following in the paragraph
> after the table that the bug resolution modifies:
>
> If the right-hand operand is any expression other than the lexical
> token ERE, the string value of the expression shall be interpreted
> as an extended regular expression, including the escape conventions
> described above. Note that these same escape conventions shall also
> be applied in determining the value of a string literal (the lexical
> token STRING), and thus shall be applied a second time when a
> string literal is used in this context.
I don't see the point of "requiring" that $0 ~ "\\f" match a FF.
Especially considering that we're going to specify $'\f'.
Many implementations don't do it and I don't imagine them
changing any time soon (several of them still don't support
{x,y} RE intervals).
I know that portably, I need RE='\\begin\{.*\}' or
RE='[\\]begin[{].*[}]' (used as $0 ~ ENVIRON["RE"]) to match on
\begin{...}. That ticket was about having POSIX stating that.
> > You said earlier that,
> >
> > echo b | awk '/[a\55c]/'
> >
> > is required to match (on ASCII-based systems), but that's not
> > the case with several implementations (like nawk on Solaris,
> > bwk's awk, mawk, FreeBSD awk...).
> >
> > Should those be considered non-compliant?
>
> The previous discussion was \056 matching any character because it
> becomes a <period>, but this case should match too (assuming both
> ASCII encoding and POSIX locale; the latter because it's a range).
[...]
Note the requirement above that $0 ~ "\\f" match on FF suggests
an implementation whose regex engine understands "\f", while
this one suggests those \x being expanded before calling the
regexp engine.
Again, in practice today one can't rely on either as
different implementations do it differently. I'm not sure
there's much value in not leaving all those unspecified given
that's it's mostly corner cases.
What problem do you see with my suggested resolution (which as
far as I can tell is compatible with most existing
implementations)?
--
Stephane
