Branch: refs/heads/blead
Home: https://github.com/Perl/perl5
Commit: 03dc0b1b4f089b6205935b158692b9364ae3453b
https://github.com/Perl/perl5/commit/03dc0b1b4f089b6205935b158692b9364ae3453b
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
UTF8_IS_DOWNGRADEABLE_START: Call less general helper
Future commits would otherwise make the expansion of this macro too
complicated for some C compilers. Use a less general internal helper
function to avoid that.
Commit: c26e6896c8580980b7d819fc5d57c410552da30c
https://github.com/Perl/perl5/commit/c26e6896c8580980b7d819fc5d57c410552da30c
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: Move macro to earlier in file
This is now defined before first use
Commit: 0176a72cd6a1944962390c37151a16b5ea9cd5e5
https://github.com/Perl/perl5/commit/0176a72cd6a1944962390c37151a16b5ea9cd5e5
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
Refactor UTF_START_MASK()
A slight change to this very low level macro (hence called a lot)
removes the need for a conditional, and causes it to work on single-byte
UTF-8 characters on ASCII platforms.
The definition is also moved to a more logical place in the file
Commit: 36da1e17a35eb23f0d666ee29dbff5c7823cad4c
https://github.com/Perl/perl5/commit/36da1e17a35eb23f0d666ee29dbff5c7823cad4c
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
M utfebcdic.h
Log Message:
-----------
utf8.h: Add a #define synonym
This is more clearly named for various uses in this file. It has an
unwieldy length, but is unlikely to be used outside it.
Commit: fcd03d925b4b3a67a6162b516b3ea4194e92bc92
https://github.com/Perl/perl5/commit/fcd03d925b4b3a67a6162b516b3ea4194e92bc92
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
M utfebcdic.h
Log Message:
-----------
utf8.h: Make a bit of EBCDIC known to ASCII
This info is needed in one other place; doing it here means only
specifying it once.
Commit: 28ca3ab57366a041138756872c2020aca0b98ec8
https://github.com/Perl/perl5/commit/28ca3ab57366a041138756872c2020aca0b98ec8
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M regcomp.c
M utf8.h
Log Message:
-----------
utf8.h: Add symbol for easing EBCDIC handling
This is then used in regcomp.c to avoid an #ifdef EBCDIC
Commit: 99904f65f9315ad1e8da23f22b561d878bcf524c
https://github.com/Perl/perl5/commit/99904f65f9315ad1e8da23f22b561d878bcf524c
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
M utfebcdic.h
Log Message:
-----------
utf8.h: Remove an EBCDIC dependency
A symbol introduced in a previous commit allows this internal macro to
only need a single version, suitable for either EBCDIC or ASCII.
Commit: 43732c4ff66897a8d76cab584f1d6c053640acd4
https://github.com/Perl/perl5/commit/43732c4ff66897a8d76cab584f1d6c053640acd4
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
M utf8.h
Log Message:
-----------
Rename internal macro and move to utf8.h
This macro has a corresponding, older, name for the non-UTF-8 case. It
makes sense to use the same paradigm, and move the definitions together
so that the comments for one don't have to be repeated for the other.
Commit: e77f0df2059f11f53c064920dbb65a20c4d18c9f
https://github.com/Perl/perl5/commit/e77f0df2059f11f53c064920dbb65a20c4d18c9f
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
M utfebcdic.h
Log Message:
-----------
utf8.h: Rmv EBCDIC dependency
This moves a #define into the common code for ASCII and EBCDIC machines.
It adds a bunch of comments about the value that I wish I hadn't had to
figure out for myself.
Commit: 7bf011a13027d25777e5d934629cb489812d20c9
https://github.com/Perl/perl5/commit/7bf011a13027d25777e5d934629cb489812d20c9
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: Add macro to compute UV skip by its log2
This macro will calculate at compile time, if passed a compile-time
constant, how many UTF-8 bytes are required to represent the parameter.
The macro is a helper which works fine except for edge cases, which a
wrapper is needed to handle.
The commit changes one instance to use this new macro
Commit: 787e83840f698af38f64833c284e0e30865698c0
https://github.com/Perl/perl5/commit/787e83840f698af38f64833c284e0e30865698c0
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M inline.h
M utf8.h
Log Message:
-----------
Reimplement OFFUNISKIP
Now that previous commits have made it fast to find the position of the
first set bit in a word, we can use a forumla to find how many bytes the
UTF-8 of that will occupy. This allows for simplification of this
macro, removing several conditionals
Commit: 03a8ddc0378fe9958d955ac7deffdfc293830b54
https://github.com/Perl/perl5/commit/03a8ddc0378fe9958d955ac7deffdfc293830b54
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: Refactor MAX_UTF8_TWO_BYTE
The previous commit removed a macro that the comments for this refer to
in explaining its derivation. So use an alternative, that is actually
clearer.
Commit: 788cdc671aaa395bf2ca2e23f659513cf6193f49
https://github.com/Perl/perl5/commit/788cdc671aaa395bf2ca2e23f659513cf6193f49
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
M utf8.h
Log Message:
-----------
utf8.h: Add new #define for extended length UTF-8
The previous commit added a convenient place to create a symbol to
indicate that the UTF-8 on this platform includes Perl's nearly-double
length extension. The platforms this isn't needed on are 32-bit ASCII
ones. This symbol allows removing one place where EBCDIC need
be considered, and future commits will use it as well.
Commit: 222d763053ff7a4e671c2a51ec02d9ec9ac7fff9
https://github.com/Perl/perl5/commit/222d763053ff7a4e671c2a51ec02d9ec9ac7fff9
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: Move all SKIP functions to be near each other
For convenient code reading
Commit: ba8e516ec8ffca4777b4e23361464ccdbec0e9ff
https://github.com/Perl/perl5/commit/ba8e516ec8ffca4777b4e23361464ccdbec0e9ff
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M embed.fnc
M embed.h
M proto.h
M utf8.c
Log Message:
-----------
utf8.c: Change name of static function
This changes only portions of the capitalization, and the new version is
more in keeping with other function names.
Commit: 4a2c769e53e2f2a1ef53ce320379b77b0f7b8f91
https://github.com/Perl/perl5/commit/4a2c769e53e2f2a1ef53ce320379b77b0f7b8f91
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
M utf8.h
Log Message:
-----------
utf8.h: Add #define
UTF_MIN_CONTINUATION_BYTE is clearer for use in some contexts
Commit: 856dd4e17cf3619ca108f22071d97cdaed91b763
https://github.com/Perl/perl5/commit/856dd4e17cf3619ca108f22071d97cdaed91b763
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
Log Message:
-----------
utf8.c: Rmv some EBCDIC dependencies
I hadn't previously noticed the underlying symmetry between the
platforms.
Commit: d49e4ce1792863bdc3e344a52aef19110508e1ac
https://github.com/Perl/perl5/commit/d49e4ce1792863bdc3e344a52aef19110508e1ac
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
Log Message:
-----------
utf8.c: Refactor internal function
The insight in the previous commit allows this function to become much
more compact.
Commit: 8b5f2733d203f1d4c77d2a7e2250f5b63c17c9e9
https://github.com/Perl/perl5/commit/8b5f2733d203f1d4c77d2a7e2250f5b63c17c9e9
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M embed.fnc
M embed.h
M proto.h
M utf8.c
Log Message:
-----------
utf8.c: Generalize static fcn
I've always been uncomfortable with the input constraints this function
had. Now that it has been refactored into using a switch(), new cases
for full generality can be added without affecting performance, and
some conditionals removed before calling it.
The function is renamed to reflect its more generality
Commit: 298e8f0e4acd3b2173e34f3bb387f4a83cd16d1b
https://github.com/Perl/perl5/commit/298e8f0e4acd3b2173e34f3bb387f4a83cd16d1b
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
Log Message:
-----------
utf8.c: Use C_ARRAY_LENGTH()
This macro is preferred to sizeof()
Commit: c5b2813428f68013e30d129e5eac9481bb0b105f
https://github.com/Perl/perl5/commit/c5b2813428f68013e30d129e5eac9481bb0b105f
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
Log Message:
-----------
utf8.c: Use STRLENs() instead of sizeof()
This makes the code easier to read.
Commit: 86fb75ad16dc25aa240885d6da2eb32c9a3685a1
https://github.com/Perl/perl5/commit/86fb75ad16dc25aa240885d6da2eb32c9a3685a1
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
Log Message:
-----------
utf8.c: Improve algorithm for detecting overflow
The code has hard-coded into it the UTF-8 for the highest representable
code point for various platforms and word sizes. The algorithm is to
compare the input sequence to verify it is <= the highest. But the tail
of each of them has some number of the highest possible continuation
byte. We need not look at the tail, as the input cannot be above the
highest possible. This commit shortens the highest string constants and
exits the loop when we get to where the tail used to be.
This change allows for the complete removal of the code that is #ifdef'd
out that would be used when we allow core to use code points up to
UV_MAX.
Commit: 7746124c78426a0d91e73706a02e5a891efde7d5
https://github.com/Perl/perl5/commit/7746124c78426a0d91e73706a02e5a891efde7d5
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
Log Message:
-----------
utf8.c: Change internal macro name
The sequences here aren't UTF-8, but UTF, since they are I8 in
UTF-EBCDIC terms
Commit: 527347e0493f4ee24d63df996fd6616806422d17
https://github.com/Perl/perl5/commit/527347e0493f4ee24d63df996fd6616806422d17
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M embed.fnc
M embed.h
M proto.h
M utf8.c
Log Message:
-----------
utf8.c: Fold 2 overlapping fcns into one
One of these functions is now only called from the other, and there is
significant overlap in their logic.
This commit refactors them into one resulting function, which is half
the code, and more straight forward.
Commit: 58b66e89ba80b4f91ff12da18da1301aa182a687
https://github.com/Perl/perl5/commit/58b66e89ba80b4f91ff12da18da1301aa182a687
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M embed.fnc
M embed.h
M proto.h
M utf8.c
Log Message:
-----------
Add helper function for longest UTF8 sequence
This specialized functionality is used to check the validity of Perl's
extended-length UTF-8, which has some ideosyncratic characteristics from
the shorter sequences. This means this function doesn't have to
consider those differences. It will be used in the next commit to avoid
some work, and to eventually enable is_utf8_char_helper() to be
simplified.
Commit: 50f7a4cea3ee6aa556d5e0312bde77e1c914e84b
https://github.com/Perl/perl5/commit/50f7a4cea3ee6aa556d5e0312bde77e1c914e84b
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M inline.h
Log Message:
-----------
inline.h: Macroize DFA for isFOO_UTF8_CHAR()
There are currently three functions for variants of finding if the next
few bytes of a string form a proper UTF-8 encoded character of some ilk.
The main code for each is identical to the others, except for the table
that drives it.
This commit makes that code a macro that takes arguments to customize
its behavior sufficiently for current and forseeable needs.
This makes it easier to keep the varieties in sync with each other with
future changes.
The macro has three exit points: 1) successful parsing
2) unsuccessful parsing
3) succesful parsing as far as it went,
but the input was exhausted before
reaching a full character.
What to do for each of these eventualities is passed to the macro. This
is a change in behavior in which 2) and 3) were not distinguished from
each other. This actually leads to fewer tests in some situations, and
future commits using this DFA for other purposes will take advantage of
it.
Commit: 3e3eb1f61aa7ee77634882c846122860a4c8ef42
https://github.com/Perl/perl5/commit/3e3eb1f61aa7ee77634882c846122860a4c8ef42
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M embed.fnc
M proto.h
M utf8.c
Log Message:
-----------
utf8.c: Change formal parameter name to fcn
This will make more sense of the next commit
Commit: e932a5edb50acc7eadabb8b398bd071556b21ded
https://github.com/Perl/perl5/commit/e932a5edb50acc7eadabb8b398bd071556b21ded
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
Log Message:
-----------
Refactor uvoffuni_to_utf8_flags_msgs
Having a fast UVOFFUNISKIP() allows this function be be refactored to
simplify it.
This commit continues to shortchange large code points and EBCDIC by a
little. For example, it checks if a 4-byte character is above Unicode,
but no 4-byte characters fit that description in UTF-EBCDIC. This will
be fixed in the next commit, which will prepare for further
enhancements.
Commit: bc658500639af2d2587b6616c7a854049ea21972
https://github.com/Perl/perl5/commit/bc658500639af2d2587b6616c7a854049ea21972
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
Log Message:
-----------
Fix EBCDIC deficiency in uvoffuni_to_utf8_flags_msgs()
Simply by adjusting the case statement labels, and adding an extra case,
the code can avoid checking for a problem on EBCDIC boxes when it would
be impossible for the problem to exist.
Commit: b3501144d975745427dbee79cd3eddd22d140c7c
https://github.com/Perl/perl5/commit/b3501144d975745427dbee79cd3eddd22d140c7c
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M embed.fnc
M proto.h
M utf8.c
Log Message:
-----------
uvoffuni_to_utf8_flags_msgs: Avoid extra conditionals
The previous commit for EBCDIC paved the way for moving some checks for
a code point being for Perl extended UTF-8 out of places where they
cannot succeed. The resultant simplifications more than compensate for
the two extra case statements added by this commit.
Commit: dc1baa353816856e1128cac9fd3bb5a21391d080
https://github.com/Perl/perl5/commit/dc1baa353816856e1128cac9fd3bb5a21391d080
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
Log Message:
-----------
utf8.c: Comment non-obvious fcn param meaning
Commit: 473bd7939eb908a550b65ad14bc58789d6f3b87c
https://github.com/Perl/perl5/commit/473bd7939eb908a550b65ad14bc58789d6f3b87c
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.c
Log Message:
-----------
utf8.c: in-line only use of two macros
These macros don't need to be macros, as they each are only called from
one place, and that isn't likely to change.
Commit: 6736ce803b72875a4cb392927e8ac445ddea2626
https://github.com/Perl/perl5/commit/6736ce803b72875a4cb392927e8ac445ddea2626
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: Reorder some preprocessor directives
This is just so that things are clearer to the reader
Commit: 02a955506b082584fee72270e4ff31b589deaacc
https://github.com/Perl/perl5/commit/02a955506b082584fee72270e4ff31b589deaacc
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utfebcdic.h
Log Message:
-----------
utfebcdic.h: White-space, comment only
Commit: bdcc1e93b12b67f35d05618013410ca92713eaf3
https://github.com/Perl/perl5/commit/bdcc1e93b12b67f35d05618013410ca92713eaf3
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: Add macros to calc UTF start byte, first cont
These two bytes are useful to know in some situations. This commit
changes a couple such places to use the first macro.
Commit: 6f8b1f9311454d2c11cb8a196b1367e9b3933cee
https://github.com/Perl/perl5/commit/6f8b1f9311454d2c11cb8a196b1367e9b3933cee
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
M utfebcdic.h
Log Message:
-----------
utf8.h: Remove EBCDIC dependency
By generalizing a macro, we can make it serve both ASCII and EBCDIC
Commit: 51b58dba7ba5a13c027c24eee220e82eab517ce5
https://github.com/Perl/perl5/commit/51b58dba7ba5a13c027c24eee220e82eab517ce5
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M charclass_invlists.h
M handy.h
M l1_char_class_tab.h
M lib/unicore/uni_keywords.pl
M regen/mk_PL_charclass.pl
M uni_keywords.h
Log Message:
-----------
Remove EBCDIC-only code
The previous commit stopped using this code, so can just get rid of it.
Commit: 6110285c6da30a8505d3b73bbbd0cbf6e0fdecac
https://github.com/Perl/perl5/commit/6110285c6da30a8505d3b73bbbd0cbf6e0fdecac
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: Move some #defines around
This moves the defines for things like surrogates, non-character code
points, etc. to a more logical order, with like adjacent to like, and
before they are otherwise used in the file.
Commit: 39fafb79d3ca9909dce0911477aa8340d9402f98
https://github.com/Perl/perl5/commit/39fafb79d3ca9909dce0911477aa8340d9402f98
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: Document some #defines
The reorganization in the previous commit revealed some undocumented
public macros
Commit: 7ce5b055d7e2c571c1f3b84b9d89acd333ba7618
https://github.com/Perl/perl5/commit/7ce5b055d7e2c571c1f3b84b9d89acd333ba7618
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
Refactor UTF8_IS_SUPER()
This uses macros recently introduced to remove an EBCDIC dependency and
make the definition simpler. It now uses the DFA, which should speed up
the non-edge case uses.
Commit: 65e4aa05ff531aa30e0844560352a4e7494f5b2d
https://github.com/Perl/perl5/commit/65e4aa05ff531aa30e0844560352a4e7494f5b2d
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: Refactor UTF8_IS_NONCHAR...
UTF8_IS_NONCHAR_GIVEN_THAT_NON_SUPER_AND_GE_PROBLEMATIC() is defined just
for backward compatability (though I don't think anyone uses it).
Swap which macro is the base level that the other is defined in terms of
Commit: 030c8206f16aeaa7a09869e51a9c4f9d85cb4c8a
https://github.com/Perl/perl5/commit/030c8206f16aeaa7a09869e51a9c4f9d85cb4c8a
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: White space only
Commit: 42b360b2e07dd1c42764c476a72cc282a4400ce9
https://github.com/Perl/perl5/commit/42b360b2e07dd1c42764c476a72cc282a4400ce9
Author: Karl Williamson <[email protected]>
Date: 2021-08-07 (Sat, 07 Aug 2021)
Changed paths:
M utf8.h
Log Message:
-----------
utf8.h: Comment changes
Compare: https://github.com/Perl/perl5/compare/306c807eed39...42b360b2e07d
