In perl.git, the branch sprout/pos has been created
<http://perl5.git.perl.org/perl.git/commitdiff/5bddf0578580e934b812d9b7eaed59fd2fbcd159?hp=0000000000000000000000000000000000000000>
at 5bddf0578580e934b812d9b7eaed59fd2fbcd159 (commit)
- Log -----------------------------------------------------------------
commit 5bddf0578580e934b812d9b7eaed59fd2fbcd159
Author: Father Chrysostomos <[email protected]>
Date: Tue Jul 23 13:15:34 2013 -0700
Stop pos() from being confused by changing utf8ness
The value of pos() is stored as a byte offset. If it is stored on a
tied variable or a reference (or glob), then the stringification could
change, resulting in pos() now pointing to a different character off-
set or pointing to the middle of a character:
$ ./perl -Ilib -le '$x = bless [], chr 256; pos $x=1; bless $x, a; print
pos $x'
2
$ ./perl -Ilib -le '$x = bless [], chr 256; pos $x=1; bless $x, "\x{1000}";
print pos $x'
Malformed UTF-8 character (unexpected end of string) in match position at
-e line 1.
0
So pos() should be stored as a character offset.
The regular expression engine expects byte offsets always, so allow it
to store bytes when possible (a pure non-magical string) but use char-
acters otherwise.
This does result in more complexity than I should like, but the alter-
native (always storing a character offset) would slow down regular
expressions, which is a big no-no.
M dump.c
M embed.fnc
M embed.h
M ext/Devel-Peek/t/Peek.t
M inline.h
M mg.c
M mg.h
M pp.c
M pp_ctl.c
M pp_hot.c
M proto.h
M regexec.c
M regexp.h
M sv.c
M sv.h
M t/op/pos.t
commit 7295ddec1a31072cceaba2a64990bad6737ca0fe
Author: Father Chrysostomos <[email protected]>
Date: Sun Jul 21 06:56:07 2013 -0700
sv.c: Remove overflow check in utf8 length cache
In order to make large values of pos() possible, the previous commit
enlarged the mg_len field of the magic struct. That field is also
used for cached utf8 length values. Since it is now large enough to
store any length the OS/compiler supports, the overflow never happens.
M sv.c
commit eee80a62990f3ac36f6d3a3c610b0ec3ed8f2006
Author: Father Chrysostomos <[email protected]>
Date: Sun Jul 21 00:38:28 2013 -0700
[perl #72766] Allow huge pos() settings
This is part of #116907, too. It also fixes #72924 as a side effect;
the next commit will explain.
The value of pos($foo) was being stored as an I32, not allowing values
above I32_MAX. Change it to SSize_t (the signed equivalent of size_t,
representing the maximum string length the OS/compiler supports).
This is accomplished by changing the size of the entry in the magic
struct, which is the simplest fix.
Other parts of the code base can benefit from this, too.
We actually cast the pos value to STRLEN (size_t) when reading
it, to allow *very* long strings. Only the value -1 is special,
meaning there is no pos. So the maximum supported offset is
2**sizeof(size_t)-2.
The regexp engine itself still cannot handle large strings, so being
able to set pos to large values is useless right now. This is but one
piece in a larger puzzle.
M MANIFEST
M mg.c
M mg.h
M pp.c
A t/bigmem/pos.t
commit decccb94ee6563dfe6edaff3ca8a605c5068bd61
Author: Father Chrysostomos <[email protected]>
Date: Sun Jul 21 00:30:20 2013 -0700
Add sv_pos_b2u_flags
This, similar to sv_pos_u2b_flags, is a more friendly variant of
sv_pos_u2b that works with 2GB strings and actually returns a
value instead of modifying a passed-in value in place through
a pointer.
The next commit will use this.
M embed.fnc
M embed.h
M proto.h
M sv.c
commit cbd5ead563d5a5df04e771f407468c0d3bcdb9f7
Author: Father Chrysostomos <[email protected]>
Date: Sat Jul 20 19:02:13 2013 -0700
perl5180delta: typo
M pod/perl5180delta.pod
-----------------------------------------------------------------------
--
Perl5 Master Repository
