Branch: refs/heads/yves/curlyx_curlym
Home: https://github.com/Perl/perl5
Commit: 015937dcd0f16a3d8facd38dab52726b185b1476
https://github.com/Perl/perl5/commit/015937dcd0f16a3d8facd38dab52726b185b1476
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M t/re/re_tests
Log Message:
-----------
t/re/re_rests - extend test to show more buffers
This is a tricky test, showing more buffers makes it a bit easier
to understand if you break it. (Guess what I did?)
Commit: 6ff80f67e955de0295d7ac13d5394c6db80b491c
https://github.com/Perl/perl5/commit/6ff80f67e955de0295d7ac13d5394c6db80b491c
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M regcomp.c
M regcomp.h
M regcomp_internal.h
M t/re/pat.t
M t/re/reg_mesg.t
Log Message:
-----------
regcomp.c - increase size of CURLY nodes so the min/max is a I32
This allows us to resolve a test inconsistency between CURLYX and CURLY
and CURLYM. We use I32 because the existing count logic uses -1 and
this keeps everything unsigned compatible.
Commit: 9509a6014a77fa0d034d4934184d1a8999ff46de
https://github.com/Perl/perl5/commit/9509a6014a77fa0d034d4934184d1a8999ff46de
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M regcomp_internal.h
M regcomp_study.c
Log Message:
-----------
regcomp_study.c - Add a way to disable CURLYX optimisations
Also break up the condition so there is one condition per line so
it is more readable, and fold repeated binary tests together. This
makes it more obvious what the expression is doing.
Commit: ca103f9971e86a5446b3bf2ab05d572068b3247c
https://github.com/Perl/perl5/commit/ca103f9971e86a5446b3bf2ab05d572068b3247c
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M regcomp_debug.c
M regcomp_study.c
M t/re/pat_re_eval.t
Log Message:
-----------
regcomp_study.c - disable CURLYX optimizations when EVAL has been seen
anywhere
Historically we disabled CURLYX optimizations when they
*contained* an EVAL, on the assumption that the optimization might
affect how many times, etc, the eval was called. However, this is
also true for CURLYX with evals *afterwards*. If the CURLYN or CURLYM
optimization can prune off the search space, then an eval afterwards
will be affected. An when you take into account GOSUB, it means that
an eval in front might be affected by an optimization after it.
So for now we disable CURLYN and CURLYM in any pattern with an EVAL.
Commit: 3d9f8117bd3ffc1edaeabee32455c16c1bcea786
https://github.com/Perl/perl5/commit/3d9f8117bd3ffc1edaeabee32455c16c1bcea786
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M regexec.c
Log Message:
-----------
regexec.c - rework CLOSE_CAPTURE() macro to take a rex argument
This allows it to be used in contexts where rex isn't set up under
this name.
Commit: ece7a67aaa55f4ef91bcc40712de819141e22e95
https://github.com/Perl/perl5/commit/ece7a67aaa55f4ef91bcc40712de819141e22e95
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M regcomp.c
M regcomp.h
Log Message:
-----------
regcomp.h - get rid of EXTRA_STEP defines
They are unused these days.
Commit: 7b755c85212c6c0c22e0275b26152d31a67c374f
https://github.com/Perl/perl5/commit/7b755c85212c6c0c22e0275b26152d31a67c374f
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M regcomp.c
Log Message:
-----------
regcomp.c - add whitespace to binary operation
The tight & is hard to read.
Commit: 6acbbb935961e7ea122571055d7b7b47027a96fe
https://github.com/Perl/perl5/commit/6acbbb935961e7ea122571055d7b7b47027a96fe
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M regcomp_trie.c
Log Message:
-----------
regcomp_trie.c - use the indirect types so we are safe to changes
We shouldnt assume that a TRIEC is a regcomp_charclass. We have a per
opcode type exactly for this type of use, so lets use it.
Commit: 15f5113cef426f0064495d18865d69304b3474a3
https://github.com/Perl/perl5/commit/15f5113cef426f0064495d18865d69304b3474a3
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M pod/perldebguts.pod
M pp_ctl.c
M regcomp.c
M regcomp.h
M regcomp.sym
M regcomp_debug.c
M regexec.c
M regexp.h
M regnodes.h
M t/re/pat.t
M t/re/pat_rt_report.t
M t/re/re_tests
Log Message:
-----------
regcomp.c - Resolve issues clearing buffers in CURLYX (MAJOR-CHANGE)
CURLYX doesn't reset capture buffers properly. It is possible
for multiple buffers to be defined at once with values from
different iterations of the loop, which doesn't make sense really.
An example is this:
"foobarfoo"=~/((foo)|(bar))+/
after this matches $1 should equal $2 and $3 should be undefined,
or $1 should equal $3 and $2 should be undefined. Prior to this
patch this would not be the case.
The solution that this patches uses is to introduce a form of
"layered transactional storage" for paren data. The existing
pair of start/end data for capture data is extended with a
start_new/end_new pair. When the vast majority of our code wants
to check if a given capture buffer is defined they first check
"start_new/end_new", if either is -1 then they fall back to
whatever is in start/end.
When a capture buffer is CLOSEd the data is written into the
start_new/end_new pair instead of the start/end pair. When a CURLYX
loop is executing and has matched something (at least one "A" in
/A*B/ -- thus actually in WHILEM) it "commits" the start_new/end_new
data by writing it into start/end. When we begin a new iteration of
the loop we clear the start_new/end_new pairs that are contained by
the loop, by setting them to -1. If the loop fails then we roll back
as we used to. If the loop succeeds we continue. When we hit an END
block we commit everything.
Consider the example above. We start off with everything set to -1.
$1 = (-1,-1):(-1,-1)
$2 = (-1,-1):(-1,-1)
$3 = (-1,-1):(-1,-1)
In the first iteration we have matched "foo" and end up with this:
$1 = (-1,-1):( 0, 3)
$2 = (-1,-1):( 0, 3)
$3 = (-1,-1):(-1,-1)
We commit the results of $2 and $3, and then clear the new data in
the beginning of the next loop:
$1 = (-1,-1):( 0, 3)
$2 = ( 0, 3):(-1,-1)
$3 = (-1,-1):(-1,-1)
We then match "bar":
$1 = (-1,-1):( 0, 3)
$2 = ( 0, 3):(-1,-1)
$3 = (-1,-1):( 3, 7)
and then commit the result and clear the new data:
$1 = (-1,-1):( 0, 3)
$2 = (-1,-1):(-1,-1)
$3 = ( 3, 7):(-1,-1)
and then we match "foo" again:
$1 = (-1,-1):( 0, 3)
$2 = (-1,-1):( 7,10)
$3 = ( 3, 7):(-1,-1)
And we then commit. We do a regcppush here as normal.
$1 = (-1,-1):( 0, 3)
$2 = ( 7,10):( 7,10)
$3 = (-1,-1):(-1,-1)
We then clear it again, but since we don't match when we regcppop
we store the buffers back to the above layout. When we finally
hit the END buffer we also do a commit as well on all buffers, including
the 0th (for the full match).
Fixes GH Issue #18865, and adds tests for it and other things.
Commit: 415eb17513636dda58df924bc76f6fc3e38d6531
https://github.com/Perl/perl5/commit/415eb17513636dda58df924bc76f6fc3e38d6531
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M MANIFEST
M t/re/regexp.t
A t/re/regexp_normal.t
Log Message:
-----------
t/re/regexp_normal.t - test "normalized" forms of patterns
This looks for discrepancies between different ways of writing
a pattern.
Commit: 8f76715e2f9352aed2f27d056e45bb09dd2e2fcb
https://github.com/Perl/perl5/commit/8f76715e2f9352aed2f27d056e45bb09dd2e2fcb
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M pod/perldebguts.pod
M regcomp.c
M regcomp.h
M regcomp.sym
M regcomp_debug.c
M regcomp_trie.c
M regexec.c
M regexp.h
M regnodes.h
M t/re/re_tests
Log Message:
-----------
regexec.c - teach BRANCH and BRANCHJ nodes to reset capture buffers
In /((a)(b)|(a))+/ we should not end up with $2 and $4 being set at
the same time. When a branch fails it should reset any capture buffers
that might be touched by its branch.
We change BRANCH and BRANCHJ to store the number of parens before the
branch, and the number of parens after the branch was completed. When
a BRANCH operation fails, we clear the buffers it contains before we
continue on.
It is a bit more complex than it should be because we have BRANCHJ
and BRANCH. (One of these days we should merge them together.)
This is also made somewhat more complex because TRIE nodes are actually
branches, and may need to track capture buffers also, at two levels.
The overall TRIE op, and for jump tries especially where we emulate
the behavior of branches. So we have to do the same clearing logic if
a trie branch fails as well.
Commit: 8cfcaa60faa46e4a349d71aa75a151d34eacd214
https://github.com/Perl/perl5/commit/8cfcaa60faa46e4a349d71aa75a151d34eacd214
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M pod/perldelta.pod
M pod/perlre.pod
M regcomp.c
M regcomp.h
M regcomp_debug.c
M regcomp_internal.h
M regcomp_study.c
M regexec.c
M regnodes.h
M t/re/pat_re_eval.t
M t/re/pat_rt_report.t
M toke.c
Log Message:
-----------
regcomp.c - add optimistic eval
This adds (*{ ... }) and (**{ ... }) as equivalents to
(?{ ... }) and (??{ ... }). The only difference being that
the star variants are "optimisitic" and are defined to never
disable optimisations. This is especially relevant now that
use of (?{ ... }) prevents important optimisations anywhere
in the pattern, instead of the older and inconsistent rules
where it only affected the parts that contained the EVAL.
It is also very useful for injecting debugging style expressions
to the pattern to understand what the regex engine is actually
doing. The older style (?{ ... }) variants would change the
regex engines behavior, meaning this was not as effective a
tool as it could have been.
Commit: 00faaca19d198f2787e106a191aad890c6c43abd
https://github.com/Perl/perl5/commit/00faaca19d198f2787e106a191aad890c6c43abd
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M regexec.c
M t/re/pat_re_eval.t
M t/re/regexp.t
Log Message:
-----------
regexec.c - fix accept in CURLYX/WHILEM construct.
The ACCEPT logic didnt know how to handle WHILEM, which for
some reason does not have a next_off defined. I am not sure why.
This was revealed by forcing CURLYX optimisations off. This includes
a patch to test what happens if we embed an eval group in the tests
run by regexp.t when run via regexp_normal.t, which disabled CURLYX ->
CURLYN and CURLYM optimisations and revealed this issue.
Commit: 7fc54939fb812cc33d7153f7cc9ebd4f979dfd5e
https://github.com/Perl/perl5/commit/7fc54939fb812cc33d7153f7cc9ebd4f979dfd5e
Author: Yves Orton <[email protected]>
Date: 2023-01-13 (Fri, 13 Jan 2023)
Changed paths:
M pod/perldelta.pod
Log Message:
-----------
perldelta - add note about regex engine changes
capture buffer semantics should now be consistent.
Compare: https://github.com/Perl/perl5/compare/6513f007a25a...7fc54939fb81
