In perl.git, the branch maint-5.20 has been updated <http://perl5.git.perl.org/perl.git/commitdiff/67c7ec2b8ce1a245a649db768794ba3997700dad?hp=51ff649d8e91721511ab3cd3c0b9cadcbf01f174>
- Log ----------------------------------------------------------------- commit 67c7ec2b8ce1a245a649db768794ba3997700dad Author: David Mitchell <[email protected]> Date: Wed Jul 2 17:13:45 2014 +0100 s///e on tainted utf8 strings got pos() messed up RT #122148: In 5.20, commit 25fdce4a165 changed the way pos() was stored in magic attached to SVs from being a byte offset to a char offset, *except* that, for efficiency, strings being used for pattern matching were kept as byte offsets (with a flag indicating thus), *except* where the SV already had magic attached (such as taint, as in the bug report and in this commit's test), in which case it kept it as chars. The code that updated pos() after an iteration of s///e was faulty: the string buffer it used for converting byte legnths to char lengths (via utf8_length()) was the wrong buffer: rather than using the src string being matched against, it was using the destination string being built up via iterations of s///. Once double-byte utf8 chars were involved, all the pos() calculations went wrong, and utf8 warnings started mysteriously appearing. (cherry picked from commit cda67c9995c6d90b71a0939aaae084e1869b8248) M pp_ctl.c M t/op/utftaint.t commit c446d1beb4cdc7574346aa8603d8b3a4a1a1d157 Author: David Mitchell <[email protected]> Date: Thu Jun 5 15:03:32 2014 +0100 when unCOWing a string, set SvCUR to 0 When a COW string is unCOWed, as well as setting SvPVX to NULL and SvLEN to 0, set SvCUR to 0 too. This is to avoid a later SvGROW on the same using the old SvCUR() value to calculate a roundup to the buffer size. Consider the following code: use Devel::Peek; for (1..3) { my $t; my $s = 'x' x 100; $t = $s; Dump $s; } Looking at the LEN line of the Dump output, we got on 5.20.0: LEN = 102 LEN = 135 LEN = 135 and after this commit, LEN = 102 LEN = 102 LEN = 102 As well as wasting space, this extra LEN was then triggering the 'skip COW if LEN >> CUR' mechanism, causing extra copies. See: [perl #121977] COWification seems expensive in PADMY variables (cherry picked from commit a7ab896004fe7cc32eeddadf760d0829e9fed13d) M sv.c commit 0ceb7d1bf322c10e668948d949f349044ca8d9fa Author: Yves Orton <[email protected]> Date: Thu May 29 11:33:56 2014 +0200 temporary fix for [perl #121975] COW speedup lost after e8c6a474 Disable use of Perl_safesysmalloc_size by default. Only use it when PERL_USE_MALLOC_SIZE is defined. Using Perl_safesysmalloc_size() perl cannot tell a deliberately preallocated buffer which we probably dont want to COW from a malloc() preallocated buffer where we probably dont care. Hopefully this fixes the slowdown observed on some platforms. (cherry picked from commit ce861ea7963084a7fe72fbb5485e77bb85037b55) M sv.c commit d3d804ae257f8e8cd44fd1950f51aa984ef6927d Author: H.Merijn Brand <[email protected]> Date: Wed Jun 11 09:45:50 2014 +0200 [perl #121585] Configure shouldn't write source tree (cherry picked from commit d611994aadec72a5ecb074ea6ec1fb111f5af31e) M Configure commit 41c493fb61d2d506a172105f3974f1878aa40389 Author: Tony Cook <[email protected]> Date: Wed Jun 11 14:52:26 2014 +1000 Andrew Fresh is now a perl author (cherry picked from commit e61ccd964839921b5f9ec9d1e9d648c4b24926a9) M AUTHORS commit 28342cfb601f39ea4d822774fe0a627ba6918153 Author: Andrew Fresh <[email protected]> Date: Sun Jun 1 11:00:39 2014 -0700 Correct architectures and versions for hints Over time, OpenBSD has kept this up-to-date locally without enough patches going upstream. This should bring perl in sync with reality. (cherry picked from commit 2b5735951688747a8a87926ebed2ae97609b0142) M hints/openbsd.sh commit f1476c064569f1f0fe409821970bf80574c99a10 Author: H.Merijn Brand <[email protected]> Date: Tue Jul 15 13:12:00 2014 +0100 evat => eval as found by Tom Hukins (cherry picked from commit b488de02fc2144db024373ef65e4e25af3f350eb) M Configure commit b32331e0335b4ec396ee50eb9e3599e83f4200d4 Author: Craig A. Berry <[email protected]> Date: Thu May 29 16:46:19 2014 -0500 Give caretx.c an explicit rule in vms/descrip_mms.template. Without an explicit rule it gets build with CFLAGS rather than CORECFLAGS, which could make it miss important things, such as PERL_CORE being defined. (cherry picked from commit 878cc5f0586e69cae0c78f759b8790bfd7c07291) M vms/descrip_mms.template commit 933ca0e7222c2b45507234766a316f6fe70255f3 Author: Jarkko Hietaniemi <[email protected]> Date: Thu May 29 15:16:47 2014 -0400 Get rid of "grep empty(sub)expression" noise. In OS X (and assumedly *BSD) Configure output starts with: grep empty(sub)expression apparently because plain 'grep' understands the 'os\(/\|\)2' somewhat differently (BRE vs ERE, RTFM re_format(7)) Cure: redirect the stderr of grep to stdout (which is then promptly redirected to /dev/null). The grep will still fail, and not take the OS/2 (or DJGPP) branch. (cherry picked from commit 48a72c671eb9f7f8ec040f42bacfe62436e37ee1) M Configure commit 60b3c17dc106b7c7b8903b44532ae0e76c17a442 Author: Brian Fraser <[email protected]> Date: Thu May 29 12:58:52 2014 +0200 Fix for [perl #121963] 5.20.0-RC1 can't compile on Android On some native android builds, this was causing warnings and possibly build errors before Config.pm was generated. (cherry picked from commit e7d73da9c2d808c39145a762f6b4146a6d17b6b5) M dist/PathTools/Cwd.pm M dist/PathTools/lib/File/Spec.pm M dist/PathTools/lib/File/Spec/Cygwin.pm M dist/PathTools/lib/File/Spec/Epoc.pm M dist/PathTools/lib/File/Spec/Functions.pm M dist/PathTools/lib/File/Spec/Mac.pm M dist/PathTools/lib/File/Spec/OS2.pm M dist/PathTools/lib/File/Spec/Unix.pm M dist/PathTools/lib/File/Spec/VMS.pm M dist/PathTools/lib/File/Spec/Win32.pm commit 2f907b45a2d9aee44168a945197b02880c2907c6 Author: Brian Fraser <[email protected]> Date: Tue Jul 15 13:08:53 2014 +0100 make regen for uconfig.h (cherry picked from commit 2473ac0469b66d799ee3e5bb3ca2cc722cb4a7a3) M uconfig.h commit 169cd05b532791531bfccc455761fb5093c75b0f Author: Brian Fraser <[email protected]> Date: Thu May 29 12:54:59 2014 +0200 SH_PATH should be $targetsh, not $sh Due to a small oversight, this was reverted in the last metaconfig update, and was preventing backticks/system/exec/two-arg-pipe-open from working in some Android devices. (cherry picked from commit 0b0a0724169ff8771b00d92da1f2d8745c9eceb8) M config_h.SH ----------------------------------------------------------------------- Summary of changes: AUTHORS | 1 + Configure | 24 ++++++++++++------------ config_h.SH | 2 +- dist/PathTools/Cwd.pm | 5 +++-- dist/PathTools/lib/File/Spec.pm | 2 +- dist/PathTools/lib/File/Spec/Cygwin.pm | 2 +- dist/PathTools/lib/File/Spec/Epoc.pm | 2 +- dist/PathTools/lib/File/Spec/Functions.pm | 2 +- dist/PathTools/lib/File/Spec/Mac.pm | 2 +- dist/PathTools/lib/File/Spec/OS2.pm | 2 +- dist/PathTools/lib/File/Spec/Unix.pm | 2 +- dist/PathTools/lib/File/Spec/VMS.pm | 2 +- dist/PathTools/lib/File/Spec/Win32.pm | 2 +- hints/openbsd.sh | 14 +++++++------- pp_ctl.c | 4 ++-- sv.c | 12 +++++++++--- t/op/utftaint.t | 11 ++++++++++- uconfig.h | 2 +- vms/descrip_mms.template | 2 ++ 19 files changed, 57 insertions(+), 38 deletions(-) diff --git a/AUTHORS b/AUTHORS index 29573b3..8aed4e2 100644 --- a/AUTHORS +++ b/AUTHORS @@ -84,6 +84,7 @@ Andrew Bettison <[email protected]> Andrew Burt <[email protected]> Andrew Cohen <[email protected]> andrew deryabin <[email protected]> +Andrew Fresh <[email protected]> Andrew Hamm <[email protected]> Andrew M. Langmead <[email protected]> Andrew Pimlott <[email protected]> diff --git a/Configure b/Configure index 293ef61..4154d7a 100755 --- a/Configure +++ b/Configure @@ -28,7 +28,7 @@ # See Porting/pumpkin.pod for more information on metaconfig. # -# Generated on Wed Apr 9 12:24:24 CEST 2014 [metaconfig 3.5 PL0] +# Generated on Wed Jun 4 08:58:13 CEST 2014 [metaconfig 3.5 PL0] # (with additional metaconfig patches by [email protected]) cat >c1$$ <<EOF @@ -91,7 +91,7 @@ esac : Proper separator for the PATH environment variable p_=: : On OS/2 this directory should exist if this is not floppy only system ":-]" -if test -d c:/. || ( uname -a | grep -i 'os\(/\|\)2' ) 2>&1 >/dev/null ; then +if test -d c:/. || ( uname -a | grep -i 'os\(/\|\)2' 2>&1 ) 2>&1 >/dev/null ; then if test -n "$OS2_SHELL"; then p_=\; PATH=`cmd /c "echo %PATH%" | tr '\\\\' / ` @@ -17748,13 +17748,13 @@ freebsd) h_shm=false val="$undef" set shmctl d_shmctl - evat $setvar + eval $setvar set shmget d_shmget - evat $setvar + eval $setvar set shmat d_shmat - evat $setvar + eval $setvar set shmdt d_shmdt - evat $setvar + eval $setvar ;; esac ;; @@ -22400,8 +22400,8 @@ find_extensions=' Scalar/List/Utils) this_ext="List/Util" ;; PathTools) this_ext="Cwd" ;; esac; - echo " $xs_extensions $nonxs_extensions" > $$.tmp; - if $contains " $this_ext " $$.tmp; then + echo " $xs_extensions $nonxs_extensions" > $tdir/$$.tmp; + if $contains " $this_ext " $tdir/$$.tmp; then echo >&4; echo "Duplicate directories detected for extension $xxx" >&4; echo "Configure cannot correctly recover from this - shall I abort?" >&4; @@ -22419,15 +22419,15 @@ find_extensions=' esac; echo "Ok. You will need to correct config.sh before running make." >&4; fi; - $ls -1 $xxx > $$.tmp; - if $contains "\.xs$" $$.tmp > /dev/null 2>&1; then + $ls -1 $xxx > $tdir/$$.tmp; + if $contains "\.xs$" $tdir/$$.tmp > /dev/null 2>&1; then xs_extensions="$xs_extensions $this_ext"; - elif $contains "\.c$" $$.tmp > /dev/null 2>&1; then + elif $contains "\.c$" $tdir/$$.tmp > /dev/null 2>&1; then xs_extensions="$xs_extensions $this_ext"; elif $test -d $xxx; then nonxs_extensions="$nonxs_extensions $this_ext"; fi; - $rm -f $$.tmp; + $rm -f $tdir/$$.tmp; ;; esac; done' diff --git a/config_h.SH b/config_h.SH index 640d0a2..fd46735 100755 --- a/config_h.SH +++ b/config_h.SH @@ -3129,7 +3129,7 @@ sed <<!GROK!THIS! >$CONFIG_H -e 's!^#undef\(.*/\)\*!/\*#define\1 \*!' -e 's!^#un * /bin/pdksh, /bin/ash, /bin/bash, or even something such as * D:/bin/sh.exe. */ -#define SH_PATH "$sh" /**/ +#define SH_PATH "$targetsh" /**/ /* SIG_NAME: * This symbol contains a list of signal names in order of diff --git a/dist/PathTools/Cwd.pm b/dist/PathTools/Cwd.pm index f27a3a0..01393f3 100644 --- a/dist/PathTools/Cwd.pm +++ b/dist/PathTools/Cwd.pm @@ -171,7 +171,7 @@ use strict; use Exporter; use vars qw(@ISA @EXPORT @EXPORT_OK $VERSION); -$VERSION = '3.47'; +$VERSION = '3.48'; my $xs_version = $VERSION; $VERSION =~ tr/_//; @@ -356,7 +356,8 @@ if ($^O =~ /android/) { $pwd_cmd = "$Config::Config{targetsh} -c pwd" } else { - $pwd_cmd = "$Config::Config{sh} -c pwd" + my $sh = $Config::Config{sh} || (-x '/system/bin/sh' ? '/system/bin/sh' : 'sh'); + $pwd_cmd = "$sh -c pwd" } } diff --git a/dist/PathTools/lib/File/Spec.pm b/dist/PathTools/lib/File/Spec.pm index 3d07687..bf0a327 100644 --- a/dist/PathTools/lib/File/Spec.pm +++ b/dist/PathTools/lib/File/Spec.pm @@ -3,7 +3,7 @@ package File::Spec; use strict; use vars qw(@ISA $VERSION); -$VERSION = '3.47'; +$VERSION = '3.48'; $VERSION =~ tr/_//; my %module = (MacOS => 'Mac', diff --git a/dist/PathTools/lib/File/Spec/Cygwin.pm b/dist/PathTools/lib/File/Spec/Cygwin.pm index c356884..a791a2a 100644 --- a/dist/PathTools/lib/File/Spec/Cygwin.pm +++ b/dist/PathTools/lib/File/Spec/Cygwin.pm @@ -4,7 +4,7 @@ use strict; use vars qw(@ISA $VERSION); require File::Spec::Unix; -$VERSION = '3.47'; +$VERSION = '3.48'; $VERSION =~ tr/_//; @ISA = qw(File::Spec::Unix); diff --git a/dist/PathTools/lib/File/Spec/Epoc.pm b/dist/PathTools/lib/File/Spec/Epoc.pm index a7ad3fb..a7859c5 100644 --- a/dist/PathTools/lib/File/Spec/Epoc.pm +++ b/dist/PathTools/lib/File/Spec/Epoc.pm @@ -3,7 +3,7 @@ package File::Spec::Epoc; use strict; use vars qw($VERSION @ISA); -$VERSION = '3.47'; +$VERSION = '3.48'; $VERSION =~ tr/_//; require File::Spec::Unix; diff --git a/dist/PathTools/lib/File/Spec/Functions.pm b/dist/PathTools/lib/File/Spec/Functions.pm index 24c88a8..0170843 100644 --- a/dist/PathTools/lib/File/Spec/Functions.pm +++ b/dist/PathTools/lib/File/Spec/Functions.pm @@ -5,7 +5,7 @@ use strict; use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION); -$VERSION = '3.47'; +$VERSION = '3.48'; $VERSION =~ tr/_//; require Exporter; diff --git a/dist/PathTools/lib/File/Spec/Mac.pm b/dist/PathTools/lib/File/Spec/Mac.pm index 3568d8d..a8dc2df 100644 --- a/dist/PathTools/lib/File/Spec/Mac.pm +++ b/dist/PathTools/lib/File/Spec/Mac.pm @@ -4,7 +4,7 @@ use strict; use vars qw(@ISA $VERSION); require File::Spec::Unix; -$VERSION = '3.47'; +$VERSION = '3.48'; $VERSION =~ tr/_//; @ISA = qw(File::Spec::Unix); diff --git a/dist/PathTools/lib/File/Spec/OS2.pm b/dist/PathTools/lib/File/Spec/OS2.pm index a8eca69..df458c9 100644 --- a/dist/PathTools/lib/File/Spec/OS2.pm +++ b/dist/PathTools/lib/File/Spec/OS2.pm @@ -4,7 +4,7 @@ use strict; use vars qw(@ISA $VERSION); require File::Spec::Unix; -$VERSION = '3.47'; +$VERSION = '3.48'; $VERSION =~ tr/_//; @ISA = qw(File::Spec::Unix); diff --git a/dist/PathTools/lib/File/Spec/Unix.pm b/dist/PathTools/lib/File/Spec/Unix.pm index 3d62478..e4eddbb 100644 --- a/dist/PathTools/lib/File/Spec/Unix.pm +++ b/dist/PathTools/lib/File/Spec/Unix.pm @@ -3,7 +3,7 @@ package File::Spec::Unix; use strict; use vars qw($VERSION); -$VERSION = '3.47'; +$VERSION = '3.48'; my $xs_version = $VERSION; $VERSION =~ tr/_//; diff --git a/dist/PathTools/lib/File/Spec/VMS.pm b/dist/PathTools/lib/File/Spec/VMS.pm index 24052e6..b045e27 100644 --- a/dist/PathTools/lib/File/Spec/VMS.pm +++ b/dist/PathTools/lib/File/Spec/VMS.pm @@ -4,7 +4,7 @@ use strict; use vars qw(@ISA $VERSION); require File::Spec::Unix; -$VERSION = '3.47'; +$VERSION = '3.48'; $VERSION =~ tr/_//; @ISA = qw(File::Spec::Unix); diff --git a/dist/PathTools/lib/File/Spec/Win32.pm b/dist/PathTools/lib/File/Spec/Win32.pm index e26479c..352ec99 100644 --- a/dist/PathTools/lib/File/Spec/Win32.pm +++ b/dist/PathTools/lib/File/Spec/Win32.pm @@ -5,7 +5,7 @@ use strict; use vars qw(@ISA $VERSION); require File::Spec::Unix; -$VERSION = '3.47'; +$VERSION = '3.48'; $VERSION =~ tr/_//; @ISA = qw(File::Spec::Unix); diff --git a/hints/openbsd.sh b/hints/openbsd.sh index fd29c71..2c24c23 100644 --- a/hints/openbsd.sh +++ b/hints/openbsd.sh @@ -44,7 +44,7 @@ esac # ARCH=`arch | sed 's/^OpenBSD.//'` case "${ARCH}-${osvers}" in -alpha-2.[0-8]|mips-2.[0-8]|powerpc-2.[0-7]|m88k-*|hppa-*|vax-*) +alpha-2.[0-8]|mips-2.[0-8]|powerpc-2.[0-7]|m88k-[2-4].*|m88k-5.[0-2]|hppa-3.[0-5]|vax-*) test -z "$usedl" && usedl=$undef ;; *) @@ -93,12 +93,12 @@ d_suidsafe=$define # cc is gcc so we can do better than -O # Allow a command-line override, such as -Doptimize=-g -case ${ARCH} in -m88k) - optimize='-O0' +case "${ARCH}-${osvers}" in +hppa-3.3|m88k-2.*|m88k-3.[0-3]) + test "$optimize" || optimize='-O0' ;; -hppa) - optimize='-O0' +m88k-3.4) + test "$optimize" || optimize='-O1' ;; *) test "$optimize" || optimize='-O2' @@ -123,7 +123,7 @@ $define|true|[yY]*) esac case "$osvers" in [012].*|3.[0-6]) - # Broken at least up to OpenBSD 3.6, we'll see about 3.7 + # Broken up to OpenBSD 3.6, fixed in OpenBSD 3.7 d_getservbyname_r=$undef ;; esac esac diff --git a/pp_ctl.c b/pp_ctl.c index 380a7fe..39b7941 100644 --- a/pp_ctl.c +++ b/pp_ctl.c @@ -319,8 +319,8 @@ PP(pp_substcont) if (!(mg = mg_find_mglob(sv))) { mg = sv_magicext_mglob(sv); } - assert(SvPOK(dstr)); - MgBYTEPOS_set(mg, sv, SvPVX(dstr), m - orig); + assert(SvPOK(sv)); + MgBYTEPOS_set(mg, sv, SvPVX(sv), m - orig); } if (old != rx) (void)ReREFCNT_inc(rx); diff --git a/sv.c b/sv.c index 25b80d6..021385c 100644 --- a/sv.c +++ b/sv.c @@ -1581,14 +1581,19 @@ Perl_sv_grow(pTHX_ SV *const sv, STRLEN newlen) newlen++; #endif +#if defined(PERL_USE_MALLOC_SIZE) && defined(Perl_safesysmalloc_size) +#define PERL_UNWARANTED_CHUMMINESS_WITH_MALLOC +#endif + if (newlen > SvLEN(sv)) { /* need more room? */ STRLEN minlen = SvCUR(sv); minlen += (minlen >> PERL_STRLEN_EXPAND_SHIFT) + 10; if (newlen < minlen) newlen = minlen; -#ifndef Perl_safesysmalloc_size - if (SvLEN(sv)) +#ifndef PERL_UNWARANTED_CHUMMINESS_WITH_MALLOC + if (SvLEN(sv)) { newlen = PERL_STRLEN_ROUNDUP(newlen); + } #endif if (SvLEN(sv) && s) { s = (char*)saferealloc(s, newlen); @@ -1600,7 +1605,7 @@ Perl_sv_grow(pTHX_ SV *const sv, STRLEN newlen) } } SvPV_set(sv, s); -#ifdef Perl_safesysmalloc_size +#ifdef PERL_UNWARANTED_CHUMMINESS_WITH_MALLOC /* Do this here, do it once, do it right, and then we will never get called back into sv_grow() unless there really is some growing needed. */ @@ -5032,6 +5037,7 @@ S_sv_uncow(pTHX_ SV * const sv, const U32 flags) } # endif SvPV_set(sv, NULL); + SvCUR_set(sv, 0); SvLEN_set(sv, 0); if (flags & SV_COW_DROP_PV) { /* OK, so we don't need to copy our buffer. */ diff --git a/t/op/utftaint.t b/t/op/utftaint.t index df99c8d..d734927 100644 --- a/t/op/utftaint.t +++ b/t/op/utftaint.t @@ -18,7 +18,7 @@ sub tainted ($) { } require './test.pl'; -plan(tests => 3*10 + 3*8 + 2*16 + 2); +plan(tests => 3*10 + 3*8 + 2*16 + 3); my $arg = $ENV{PATH}; # a tainted value use constant UTF8 => "\x{1234}"; @@ -149,3 +149,12 @@ for my $ary ([ascii => 'perl'], [latin1 => "\xB6"]) { 'ok', {switches => ["-T", "-l"]}, "therefore swash_init should be taint agnostic"); } + +{ + # RT #122148: s///e on tainted utf8 strings got pos() messed up in 5.20 + + my @p; + my $s = "\x{100}\x{100}\x{100}\x{100}". $^X; + $s =~ s/\x{100}/push @p, pos($s); "xxxx";/eg; + is("@p", "0 1 2 3", "RT #122148"); +} diff --git a/uconfig.h b/uconfig.h index a2cb550..e5cca08 100644 --- a/uconfig.h +++ b/uconfig.h @@ -4718,6 +4718,6 @@ #endif /* Generated from: - * 986d2349e078df67f02d5b9747b7fe65abb0d2875fb52a836138919aafe08545 config_h.SH + * 7557e985de18f71e80f627226b454bc8eaf20477dcf0c45b5b2c51ec792f5c89 config_h.SH * dbc8d38ba52ae23e5423418bb3f56b1b6fcdaa82cf71ba0be3463e8221bfe0c0 uconfig.sh * ex: set ro: */ diff --git a/vms/descrip_mms.template b/vms/descrip_mms.template index 94e75ae..69a898c 100644 --- a/vms/descrip_mms.template +++ b/vms/descrip_mms.template @@ -665,6 +665,8 @@ $(ARCHAUTO)time.stamp : # AUTOMATICALLY GENERATED MAKE DEPENDENCIES--PUT NOTHING BELOW THIS LINE av$(O) : av.c $(h) $(CC) $(CORECFLAGS) $(MMS$SOURCE) +caretx$(O) : caretx.c $(h) + $(CC) $(CORECFLAGS) $(MMS$SOURCE) deb$(O) : deb.c $(h) $(CC) $(CORECFLAGS) $(MMS$SOURCE) doio$(O) : doio.c $(h) -- Perl5 Master Repository
