This and other RFCs are available on the web at http://dev.perl.org/rfc/ =head1 TITLE Regex: Support for incremental pattern matching =head1 VERSION Maintainer: Damian Conway <[EMAIL PROTECTED]> Date: 11 August 2000 Version: 1 Mailing List: [EMAIL PROTECTED] Number: 93 =head1 ABSTRACT This RFC proposes that, in addition to strings, subroutine references may be bound (with =~ or !~ or implicitly) to a regular expression. =head1 DESCRIPTION It is proposed that the Perl 6 regular expression engine be extended to allow a regex to match against an incremental character source, rather than only against a fixed string. Specifically, it is proposed that a subroutine reference could be bound to a regular expression: sub {...} =~ /pattern/; As the regular expression is matched, it would make calls to the subroutine to request additional characters to match, or (after it has matched) to return any unused characters. When the regex engine requires additional characters to match, the subroutine would be called with a single argument, and would be expected to return a character string containing the extra characters. The single argument would specify how many characters should be returned (typically this would be 1, unless internal analysis by the regex engine can deduce that more than one character will be required). Returning fewer than the requested number of characters would typically indicate a premature end-of-string and would probably trigger backtracking and/or failure to match. When the match is finished, the subroutine would be called one final time, and passed two arguments: a string containing the "unused" characters (what would be $' for a fixed string), and a flag set to 1. The subroutine could use this call to push-back (or cache) unused data. In the case of a failure to match (or success of the !~ operator), every character requested during the match would be sent back. A typical structure for a subroutine against which a regex was matched would therefore be: sub s { if ($_[1]) { # "putback unused data" request recache($_[0]); } else { # "send more data" request return get_chars(max=>$_[0]) } } =head2 Examples The most obvious example would be matching against an input stream: sub { $_[1] ? $fh->pushback($_[0]) : $fh->getn($_[0]) } =~ /pat/; which could also be written: ^1 ? $fh->pushback(^0) : $fh->getn(^0) =~ /pat/; Of course, it would often be useful to have a subroutine that returns a closure on a particular filehandle: sub fhmatch { ^1 ? $_[0]->pushback(^0) : $_[0]->getn(^0) } fhmatch($fh) =~ /pat/ fhmatch(\*STDIN) =~ /pat/ # etc. In fact, this might be so commonly useful that matching against a file handle should be made to work directly. That is: $fh =~ /pat/ \*STDIN =~ /pat/ One could then do interactive lexing cleanly: until (eof $fh) { switch ($fh) { /^\s*/; # skip leading whitespace case /^(lexeme1)/ { push @tokens, $1=>LEX1 } case /^(lexeme2)/ { interact_somehow } case /^(lexeme3)/ { push @tokens, $1=>LEX3 } # etc. } } Note the use of the proposed PAIR data structure to store tokens in the above example. Because the character source is a subroutine, one could also match against data coming out of a socket: my $cache = ""; sub matching_socks { if ($_[1]) { $cache .= $_[0]; return } # putback if (length($cache) < $_[0]) { # not enough cached my $extra; # so get some more recv(SOCKET, $extra, $_[0]-length($cache)); $cache .= $extra; } return substr($cache,0,$_[0],""); } switch (\&matching_socks) { case /pat1/ { action1() } case /pat2/ { action1() } case /pat3/ { action1() } #etc. } or any other source: sub mega_ape { return join "", map {['a'..'z',(' ')x6]->[rand 32]} (1..$_[0]) unless $_[1] } \&mega_ape =~ /Now is the Winter of our discontent.../i; print "Art after ", length($`), "chars\n"; =head1 IMPLEMENTATION Dammit, Jim, I'm a doctor, not an magician! Probably needs to be integrated with IO disciplines too. =head1 REFERENCES RFC 22 : Builtin switch statement RFC 23 : Higher order functions RFC 84 : Replace => (stringifying comma) with => (pair constructor)
