Branch: refs/heads/smoke-me/khw-safe
Home: https://github.com/Perl/perl5
Commit: 41601833d624501369c07642b272a219c0579364
https://github.com/Perl/perl5/commit/41601833d624501369c07642b272a219c0579364
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M cpan/Test-Harness/lib/App/Prove.pm
M cpan/Test-Harness/lib/App/Prove/State.pm
M cpan/Test-Harness/lib/App/Prove/State/Result.pm
M cpan/Test-Harness/lib/App/Prove/State/Result/Test.pm
M cpan/Test-Harness/lib/TAP/Base.pm
M cpan/Test-Harness/lib/TAP/Formatter/Base.pm
M cpan/Test-Harness/lib/TAP/Formatter/Color.pm
M cpan/Test-Harness/lib/TAP/Formatter/Console.pm
M cpan/Test-Harness/lib/TAP/Formatter/Console/ParallelSession.pm
M cpan/Test-Harness/lib/TAP/Formatter/Console/Session.pm
M cpan/Test-Harness/lib/TAP/Formatter/File.pm
M cpan/Test-Harness/lib/TAP/Formatter/File/Session.pm
M cpan/Test-Harness/lib/TAP/Formatter/Session.pm
M cpan/Test-Harness/lib/TAP/Harness.pm
M cpan/Test-Harness/lib/TAP/Harness/Env.pm
M cpan/Test-Harness/lib/TAP/Object.pm
M cpan/Test-Harness/lib/TAP/Parser.pm
M cpan/Test-Harness/lib/TAP/Parser/Aggregator.pm
M cpan/Test-Harness/lib/TAP/Parser/Grammar.pm
M cpan/Test-Harness/lib/TAP/Parser/Iterator.pm
M cpan/Test-Harness/lib/TAP/Parser/Iterator/Array.pm
M cpan/Test-Harness/lib/TAP/Parser/Iterator/Process.pm
M cpan/Test-Harness/lib/TAP/Parser/Iterator/Stream.pm
M cpan/Test-Harness/lib/TAP/Parser/IteratorFactory.pm
M cpan/Test-Harness/lib/TAP/Parser/Multiplexer.pm
M cpan/Test-Harness/lib/TAP/Parser/Result.pm
M cpan/Test-Harness/lib/TAP/Parser/Result/Bailout.pm
M cpan/Test-Harness/lib/TAP/Parser/Result/Comment.pm
M cpan/Test-Harness/lib/TAP/Parser/Result/Plan.pm
M cpan/Test-Harness/lib/TAP/Parser/Result/Pragma.pm
M cpan/Test-Harness/lib/TAP/Parser/Result/Test.pm
M cpan/Test-Harness/lib/TAP/Parser/Result/Unknown.pm
M cpan/Test-Harness/lib/TAP/Parser/Result/Version.pm
M cpan/Test-Harness/lib/TAP/Parser/Result/YAML.pm
M cpan/Test-Harness/lib/TAP/Parser/ResultFactory.pm
M cpan/Test-Harness/lib/TAP/Parser/Scheduler.pm
M cpan/Test-Harness/lib/TAP/Parser/Scheduler/Job.pm
M cpan/Test-Harness/lib/TAP/Parser/Scheduler/Spinner.pm
M cpan/Test-Harness/lib/TAP/Parser/Source.pm
M cpan/Test-Harness/lib/TAP/Parser/SourceHandler.pm
M cpan/Test-Harness/lib/TAP/Parser/SourceHandler/Executable.pm
M cpan/Test-Harness/lib/TAP/Parser/SourceHandler/File.pm
M cpan/Test-Harness/lib/TAP/Parser/SourceHandler/Handle.pm
M cpan/Test-Harness/lib/TAP/Parser/SourceHandler/Perl.pm
M cpan/Test-Harness/lib/TAP/Parser/SourceHandler/RawTAP.pm
M cpan/Test-Harness/lib/TAP/Parser/YAMLish/Reader.pm
M cpan/Test-Harness/lib/TAP/Parser/YAMLish/Writer.pm
M cpan/Test-Harness/lib/Test/Harness.pm
Log Message:
-----------
TAP::Harness: Move timer initialization
Prior to this commit, the timers for counting elapsed time and CPU usage
were begun when a job's first output appears. This yields inaccurate
results. These results are saved in t/test_state for future runs so
that they can start the longest-running tests first, which leads to
using the available cores more efficiently. (If you start a long running
test after everything else is nearly done, you have to wait for it to
finish before the suite as a whole is; if you start the long ones first,
and the shortest last, you don't have to wait very long for any
stragglers to complete.) Inaccurate results here lead to this
situation, which we were often seeing in the podcheck.t test.
The worst case is if there is heavy computation at the beginning of the
test being run. podcheck, for example, examines all the pods in the
directory structure to find which links to other pods do or do not have
corresponding anchors. Output doesn't happen until the analysis is
complete. On my system, this takes over 30 seconds, but prior to this
commit, what was noted was just the time required to do the output,
about 200 milliseconds. The result was that podcheck was viewed as
being one of the shortest tests run, so was started late in the process,
and generally held up the completion of it.
This commit by itself doesn't improve the test completion very much,
because, test tests are run a whole directory at a time, and the
directory podcheck is in, for example, is run last. The next commit
addresses that.
Commit: fcbdd1f2ea2c87ba040bd79e3a1b83f7cb0996e1
https://github.com/Perl/perl5/commit/fcbdd1f2ea2c87ba040bd79e3a1b83f7cb0996e1
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M t/harness
Log Message:
-----------
XXX env name: t/harness: Add option for faster test suite execution
This commit adds an environment variable, XXX, which if set to non-zero
increases the parallelism in the execution of the test suite, speeding
it up on systems with multiple cores.
Normally, there are two main test sections, one for core and the second
for non-core tests, and the testing of the non-core one doesn't begin
until the first is complete. Within each section, there are a number of
test categories, like 're' for regular expressions, and 'JSON::PP' for
the pure perl implementation of JSON.
Within each category, there are various single .t test files. Some
categories can have those be tested in parallel; some require them to be
done in a particular order, say because an earlier .t does setup for
subsequent ones. We already have this capability.
Completion of all the tests in a category is not needed before those of
another category can be started. This is how it already works.
However, the core section categories are ordered so that they begin in a
logical order for someone trying to get perl to work. First to start
are the basic sanity tests, then by roughly decreasing order of
widespread use in perl programs in the wild, with the final two
categories, porting and perf, being mainly of use to perl5 porters.
These two categories aren't started until all the tests in the earlier
categories are started. We have some long running tests in those two
categories, and generally they delay the start of the entire second section.
If those long running tests could be started sooner, shorter tests in
the first section could be run in parallel with them, increasing the
average CPU utilization, and the second section could begin (and hence
end) earlier, shortening the total elapsed execution time of the entire
suite.
The second section has some very long running tests. JSON-PP is one of
them. If it could run in parallel with tests from the first section,
that would also speed up the completion of the suite.
The environment variable added by this commit does both things. The
basic sanity test categories in the first section continue to be started
before anything else. But then all other tests are run in decreasing
order of elapsed time they take to run, removing the boundaries between
some categories, and between the two sections.
The gain from this increases as the number of jobs run in parallel does;
slower high core platforms have the highest increase. On the old
dromedary with 24 cores, the gain is 20%, almost 2 minutes. On my more
modern box with 12 cores, it is 8%.
Commit: 91db9186dcce85c9794d279d334f38917df519e9
https://github.com/Perl/perl5/commit/91db9186dcce85c9794d279d334f38917df519e9
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M dist/Time-HiRes/HiRes.pm
M dist/Time-HiRes/HiRes.xs
Log Message:
-----------
HiRes.xs: Remove unused macro defns
Commit: ab1d15935408f8e1d9fb774be598865b3f52a8a1
https://github.com/Perl/perl5/commit/ab1d15935408f8e1d9fb774be598865b3f52a8a1
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M locale.c
Log Message:
-----------
locale.c: Fix typo in #ifdef
This misspelling led to the code assuming that the platform didn't have
a feature that, if used, would result in faster execution.
Commit: cf8996e032103899eaa1bfa4f713db96d098b930
https://github.com/Perl/perl5/commit/cf8996e032103899eaa1bfa4f713db96d098b930
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M ext/POSIX/POSIX.xs
Log Message:
-----------
POSIX.xs: Use alternative functions if avail
These preprocessor directives failed to account for the existence of
common alternative functions (which the implementation knows about) if
the plain function is not available on the platform.
I doubt that this makes any difference, but it makes these conditionals
consistent with the others nearby.
Commit: 34b564f2f1524dccdbeb81c17456a2e4503c3620
https://github.com/Perl/perl5/commit/34b564f2f1524dccdbeb81c17456a2e4503c3620
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M ext/POSIX/POSIX.xs
Log Message:
-----------
POSIX.xs: White-space only
Indent to standards
Commit: e5dc3909b76a35bc17cd94516735d7958da4f9d8
https://github.com/Perl/perl5/commit/e5dc3909b76a35bc17cd94516735d7958da4f9d8
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M perlvars.h
Log Message:
-----------
perlvars.h: Fix comment
Commit: 19e68f94941c8deffeeab672a4982729bd26dc35
https://github.com/Perl/perl5/commit/19e68f94941c8deffeeab672a4982729bd26dc35
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M time64_config.h
Log Message:
-----------
time64_config.h: #include reentr.h
This is so it will get any reentrant versions automatically
Commit: e30ff8d1c6b0ec2a75c2a21fd9cb8b40ff97efab
https://github.com/Perl/perl5/commit/e30ff8d1c6b0ec2a75c2a21fd9cb8b40ff97efab
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M perl.h
Log Message:
-----------
perl.h: Remove ';' from 'NOOP;'
These are useless here
Commit: fe715f4fd20e03796d464b69631e02f41a321ec9
https://github.com/Perl/perl5/commit/fe715f4fd20e03796d464b69631e02f41a321ec9
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M thread.h
Log Message:
-----------
XXX Change many-reader mutex API to conform with other
Commit: 28aa495907e4ef5045dcc5b17eac6ba7abce6fff
https://github.com/Perl/perl5/commit/28aa495907e4ef5045dcc5b17eac6ba7abce6fff
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M thread.h
Log Message:
-----------
Add start-up/tear-down for many-reader mutexes
These were missed in 5640a370e8b19af74b8ca0b4694464c21a87916b.
Commit: a5bf1e2aaf64e9ed5df18dfd8e85db6d4276b027
https://github.com/Perl/perl5/commit/a5bf1e2aaf64e9ed5df18dfd8e85db6d4276b027
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M perl.h
Log Message:
-----------
perl.h: Move some code around
This is in preparation for future commits where things will be needed
earlier/later than currently.
Commit: 01635abab947a2c789b395ec7b6f0090f784e1e8
https://github.com/Perl/perl5/commit/01635abab947a2c789b395ec7b6f0090f784e1e8
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M ext/POSIX/POSIX.xs
M locale.c
M perl.h
Log Message:
-----------
Change name of mutex macro.
This macro is for localeconv(); the new name is clearer as to the
meaning, and this preps for further changes.
Commit: 5aa426459b83d8e326166c987416b12878d35152
https://github.com/Perl/perl5/commit/5aa426459b83d8e326166c987416b12878d35152
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M makedef.pl
M perl.h
M perlvars.h
Log Message:
-----------
Refactor locale mutex setup
This was prompted by my realization that even on a locale thread-safe
platform, there are functions we call that may not be thread-safe in
that they return their results in an internal static buffer, which may
be process-wide instead of per-thread. Tomasz Konojacki++ briefly
looked at Windows source code for localeconv() and this indeed did
appear to be the case.
If we thought a platform was thread-safe, no locale mutexes were set up,
and instead the calls in the code to lock were no-oops. This would lead
to potential races, the most likely candidate being localeconv(). None
have been reported, at least as far as we know. Likely that function
isn't called frequently. This would be true on both Posix 2008 and
Windows platforms, except possibly for FreeBSD, which may be the only
platform that we support that has a localeconv_l() function, which is
supposed to be immune from this issue..
The solution adopted here is to test for all the possible functions that
the Perl core uses that may be susceptible to this, and to set up the
mutex if any are found. Thus there won't be no-ops where there should
be a lock.
Commit: 3e36fe54ccf94f24b08451030a0271b809639aee
https://github.com/Perl/perl5/commit/3e36fe54ccf94f24b08451030a0271b809639aee
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M ext/POSIX/POSIX.xs
M locale.c
M perl.h
Log Message:
-----------
XXX fill in commit Name individual locale locks
These locks for different functions all use the same underlying mutex;
but that may not always be the case. By creating separate names
used only when we think they will be necessary, the compiler will
complain if the conditions in the code that actually use them are the
same. Doing this showed a misspelling in an #ifdef, fixed in XXX
Commit: d1a778521463e2c14c5fdded7912623e2e2b7471
https://github.com/Perl/perl5/commit/d1a778521463e2c14c5fdded7912623e2e2b7471
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M perl.h
M perlvars.h
Log Message:
-----------
Add capability of many-reader locale lock
This will be used in future commits
Commit: 0743ce93a9b373ad1f8f56e4f7753dfca23b8089
https://github.com/Perl/perl5/commit/0743ce93a9b373ad1f8f56e4f7753dfca23b8089
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M perl.h
M time64.c
M util.c
Log Message:
-----------
Change name of mutex macro for ENV/LOCALE locking
This combines two diffrent mutexes, and it isn't clear from the former
name what precisely is happening.
Commit: 0e7cc27dec88fcc6031bf62363a125b95e5f5c5f
https://github.com/Perl/perl5/commit/0e7cc27dec88fcc6031bf62363a125b95e5f5c5f
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M perl.h
M perlvars.h
Log Message:
-----------
perl.h: Add capability for many-reader ENV mutex locking
There are several places where there could be a problem in the
environment were changed by another thread when a function is executing,
but otherwise if another thread were reading the environment at the same
time, there isn't a problem. This adds mutex for that situation.
Future commits will take advantage of it.
Commit: 08a84ff8ef11ea1e9d56a4355c28d3446421c2b7
https://github.com/Perl/perl5/commit/08a84ff8ef11ea1e9d56a4355c28d3446421c2b7
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M os2/os2.c
Log Message:
-----------
os2: Use many reader lock instead of exclusive
This is just reading the environment, not changing it, so a many readers
can be accessing it at the same time.
Commit: af5f2d4a6c4ec57864895fac96a2fb9e5f637572
https://github.com/Perl/perl5/commit/af5f2d4a6c4ec57864895fac96a2fb9e5f637572
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M locale.c
Log Message:
-----------
XXX prob drop; done before anything so no races
Commit: 511f2ca82bc474288418630889981935e3c886d1
https://github.com/Perl/perl5/commit/511f2ca82bc474288418630889981935e3c886d1
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M perl.h
Log Message:
-----------
XXX finish msg Implement combo ENV/LOCALE mutexes
There are cases where an executing function is vulnerable to either the
locale or environment being changed by another thread. This implements
macros that use mutexes to protect these critical sections. There are
two cases, one where the functions only read; and one where they can
write, or need exclusive control so that a competing thread can't
overwrite the returned static buffer before it is safely copied.
5.32 had a placeholder for these, but didn't actually implement it.
Instead it locked just the ENV portion. On modern platforms with
thread-safe locales, the locale portion is a no-op anyway, so things
worked on them.
This new commit extends that safety to other platforms. This has long
been a vulnerability in Perl, but
Commit: c057f520c22048ab83e4e4f012fd745dcf806057
https://github.com/Perl/perl5/commit/c057f520c22048ab83e4e4f012fd745dcf806057
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M locale.c
M util.c
Log Message:
-----------
Move mutex to called function
This function has a critical section. Since it is a long-standing API
function, it needs to set one up itself rather than relying on callers
to do it. Remove the caller's set up, and do it inside.
Commit: d9039247e6e8d59f31b89968418f142cbde016a0
https://github.com/Perl/perl5/commit/d9039247e6e8d59f31b89968418f142cbde016a0
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M locale.c
M util.c
Log Message:
-----------
strftime is a critical section
with regard to changes in the locale and environment. And it writes
certain global variables, so must be write-locked.
Commit: 02ab3ea1a917dec4cdbe26275f7f81a06afed8b1
https://github.com/Perl/perl5/commit/02ab3ea1a917dec4cdbe26275f7f81a06afed8b1
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M Configure
M Cross/config.sh-arm-linux
M Cross/config.sh-arm-linux-n770
M NetWare/config.wc
M Porting/config.sh
M config_h.SH
M configure.com
M metaconfig.h
M plan9/config_sh.sample
M uconfig.h
M uconfig.sh
M uconfig64.sh
M win32/config.gc
M win32/config.vc
Log Message:
-----------
preserves
Commit: 62eb84a7f54d85572fd502aea0329512c862f2b0
https://github.com/Perl/perl5/commit/62eb84a7f54d85572fd502aea0329512c862f2b0
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M inline.h
M perl.h
Log Message:
-----------
Add GETENV_LOCK
get_env() needs to lock other threads from writing to the environment
while it is executing. It may need to have an exclusive lock if those
threads can clobber its buffer before it gets a chance to save them.
The previous commit has added a Configure probe which tells us if that
is the case. This commit uses it to select which type of mutex to use.
Commit: 6fd675a0300afc3890669ef037dc136fb719496c
https://github.com/Perl/perl5/commit/6fd675a0300afc3890669ef037dc136fb719496c
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M pod/perllocale.pod
Log Message:
-----------
perllocale: Remove stray markup
Commit: 0413737b03627a7d6515bb71f074b500156d9589
https://github.com/Perl/perl5/commit/0413737b03627a7d6515bb71f074b500156d9589
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M ext/POSIX/POSIX.xs
M perl.h
M time64.c
M util.c
M win32/win32.c
Log Message:
-----------
l fcn locks
Commit: f12f22816b64c8a62565b0910e6b8304624993f8
https://github.com/Perl/perl5/commit/f12f22816b64c8a62565b0910e6b8304624993f8
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M perl.h
Log Message:
-----------
LC_NUMERIC on old windows
Commit: 6c381ab4eba3dc843ffb8bbd81a78130db5107f5
https://github.com/Perl/perl5/commit/6c381ab4eba3dc843ffb8bbd81a78130db5107f5
Author: Karl Williamson <[email protected]>
Date: 2020-12-01 (Tue, 01 Dec 2020)
Changed paths:
M embedvar.h
M inline.h
M intrpvar.h
M perl.h
M util.c
Log Message:
-----------
mem_log
Compare: https://github.com/Perl/perl5/compare/41601833d624%5E...6c381ab4eba3
