=head1 TITLE
Regex assertions in plain Perl code
=head1 VERSION
Maintainer: Bart Lateur <[EMAIL PROTECTED]>
Date: 28 Sep 2000
Mailing List: <[EMAIL PROTECTED]>
Number: 348
Version: 2
Status: Developing (candidate for freeze)
=head1 ABSTRACT
Likely the most justifiable reason to want to be able execute embedded
Perl code while trying to match a regex pattern, is to include some
extra tests on the data just matched. The fact that the current
implementation of the "experimental" /(?{...})/ construct, doesn't do
just that, feels like a design mistake. So the proposal is to drop the
current implementation of (?{...}) in favour of code assertions.
=head1 CHANGES
=over 2
=item *
Removed stress on deleting "local" feature
=item *
Added examples on why other "advanced" features are or aren't
indispensable
=back
=head1 DESCRIPTION
Likely the most justifiable to want to be able to execute Perl code in a
regex, is to have additional checks, to see if what you just matched is
indeed acceptable. Quite a few prominent Perl people have expressed
having been unpleasantly surprised when they found out that the current
implementation of /?{...}/ doesn't do that.
Assertions are already familiar to people writing regexes: for example,
/\b/, lookahead and lookbehind, and anchors, are all assertions, but
only in matching subpatterns, not in code. Adding the option to do
similar tests in code, seems like a powerful addition, while maintaining
the basic spirit of regexes.
The main problem with the current implementation of (?{...}) is that it
"always succeeds". In case of an assertion, it is the outcome of the
execution of the embedded code, that decides if the test succeeds
(return value is true), meaning it is safe to continue; or fails (return
value is false), in which case the regex engine should abort exploring
this branch, and immediately backtrack.
=head2 example
RFC 197 proposes a specific syntactic addition to regexes, just to check
if a number is in a specific range. This is just one of the many things
that could easily achieved using an assertion:
/(\d+)(?{$1<256})/ # proposed syntax, recycling Perl5's syntax
If your string = "1234", the subpattern /(\d+)/ initially matches all
digits, stuffs them in $1, and calls the assertion code. This code
additionally if what was matched is within range, in this case, less
than 256. If this fails, the submatch fails.
This feature might result in matching some unexpected substrings: both
"123 and "234" are acceptable matches according to this assertion. In
order only to match what you really want to match, you may have to add
some anchors, lookahead and/or lookbehind. So it will take some getting
used to. This does not make it less valuable.
=head2 Getting rid of the current (?{...}) construct
The current implementation for embedding code in regexes, is not aimed
at assertions. Instead, it's only useable for its "effect at a
distance", which is simply horrible. Basically, it simply provides a
means to pass data around between various parts of the regex. This is a
unhealty situation, since it makes regexes that use it, look incredibly
obfuscated, and it requires deep knowledge on how a match in a regex
proceeds.
In addition, it promotes changing global data structures. Since the code
is executed everytime something promising is matched, and not just after
a complete match, the code will probably be executed more often than
desired.
That is the reason why Perl5 has built-in support to I
modify global data structures, so that the effect can automatically be
undone when backtracking. This makes the implementation very tricky. I
wouldn't be surprised if precisely this feature is the main reason why
the current implementation is so notoriously unstable.
Richard Proctor even wants to go further still: in his RFC 274,
"Generalised Additions to Regexs", he proposes to add support for
executing some embedded Perl code only while backtracking, specifically
to undo changes by hand. I think that this would make the situation even
worse than it is today.
Even knowing when and why the embedded code is executed, is far from
obvious. Take this example:
$_ = "SKIP buzzer";
if(/(?{print "Testing\n";})([a-z])\1(?{print "Got a match: $1\n"})/) {
print "YES\n";
} else {
print "NO\n";
}
This prints:
Testing
Testing
Testing
Got a match: z
YES
The fact that the embedded code is called 3 times, not more, surely
suprised me. It probably will surprise many people. Apparently, it is
only executed once for every lowercase letter, not just for any
character.
This inpredictability is yet another reason to discourage incrementally
modify global data structures.
=head2 enter assertions
The above considerations, which are annoying at least for executing
embedded generic code, are not hindrances at all for processing
assertions.
In spirit, assertions are intended to have a local effect only, which is
only used immediately to direct p
