Attached is a comment that captures a lot of the complexities of case
mapping. It's a bit verbose, but it seems worthwhile to get this
information recorded somewhere. Includes examples and tests to cover
those examples.
Regards,
Jeff Davis
From d8546716261bf3c58cd111670f860e26ddd76b8f Mon Sep 17 00:00:00 2001
From: Jeff Davis <[email protected]>
Date: Mon, 24 Aug 2026 11:29:46 -0700
Subject: [PATCH v1] Add "Case Mapping Complexities" comment and test.
There are a number of important edge cases and complexities that some
callers need to consider, e.g. string expansion. Also add tests to
cover the examples in the comment.
---
src/backend/utils/adt/pg_locale.c | 68 ++++++++++++++++++++++
src/test/regress/expected/collate.utf8.out | 8 ++-
src/test/regress/sql/collate.utf8.sql | 4 +-
3 files changed, 77 insertions(+), 3 deletions(-)
diff --git a/src/backend/utils/adt/pg_locale.c b/src/backend/utils/adt/pg_locale.c
index 7f73cd75956..3b9c9524315 100644
--- a/src/backend/utils/adt/pg_locale.c
+++ b/src/backend/utils/adt/pg_locale.c
@@ -1317,6 +1317,66 @@ strupper_c(char *dst, size_t dstsize, const char *src, size_t srclen)
return srclen;
}
+/*
+ * Case Mapping Complexities
+ *
+ * Below are general notes on the complexities of Unicode case mapping (the C
+ * locale uses simple ASCII semantics). See the Unicode Standard and the
+ * provider implementation for details.
+ *
+ * Case mapping can depend on the provider and locale, but in practice there
+ * are three things that matter:
+ *
+ * 1. On what version of Unicode is the provider based?
+ * 2. Is the locale based on "simple" (libc locales & C.UTF-8) or "full" (ICU
+ * locales & PG_UNICODE_FAST) case mappings?
+ * 3. Is the language one of "tr" or "az" (which have special rules for "i")?
+ *
+ * There are very few differences in case mapping among different languages
+ * and regions. One important difference, though: if the language is "az" or
+ * "tr", the uppercase of "i" is U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE,
+ * and the lowercase/foldcase of "I" is U+0131 LATIN SMALL LETTER DOTLESS I.
+ *
+ * Case mapping is for both Case Conversion (lowercasing, titlecasing, and
+ * uppercasing) as well as Case Folding. Case Conversion is for display;
+ * whereas Case Folding is to create a canonical caseless form of the string
+ * for case-insensitive matching.
+ *
+ * For "full" case mapping, some characters map to more than one other
+ * character, so the result may be longer than the original. Unicode defines
+ * the maximum string expansion to be 3x the code points (not to be confused
+ * with bytes); see Unicode 17.0 section 5.18.2. Examples: U+0130 LATIN
+ * CAPITAL LETTER I WITH DOT ABOVE lowercases to "i" followed by U+0307
+ * COMBINING DOT ABOVE; U+FB01 LATIN SMALL LIGATURE FI titlecases to "Fi";
+ * U+0390 GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS uppercases to <0399
+ * 0308 0301>; U+00DF LATIN SMALL LETTER SHARP S casefolds to "ss". NB: even
+ * for "simple" case mapping, the byte length can change because the mapped
+ * characters may have a different encoded length.
+ *
+ * Some characters have more than two forms. For instance, Greek Sigma has
+ * forms U+03A3 GREEK CAPITAL LETTER SIGMA, U+03C3 GREEK SMALL LETTER SIGMA,
+ * and U+03C2 GREEK SMALL LETTER FINAL SIGMA.
+ *
+ * When converting case, the mapping chosen may depend on the context within
+ * the string; i.e. it's not always a pure mapping. For instance, lowercasing
+ * U+03A3 results in U+03C2 if it's at the end of a word; otherwise U+03C3.
+ * Titlecasing uses uppercase (or titlecase, if available) mappings for the
+ * initial letter of a word; otherwise it uses the lowercase mapping.
+ * Casefolding is a pure mapping and never depends on context.
+ *
+ * Normalization is useful after casefolding to improve the quality of
+ * caseless matching. For a single edge case, U+0345 COMBINING GREEK
+ * YPOGEGRAMMENI, casefolding maps to a character with a different combining
+ * class, and it's useful to normalize both before and after casefolding.
+ *
+ * Mappings may depend on the provider and the version of Unicode on which it
+ * is based. Unassigned code points map to themselves, so their mapping may
+ * change in a later version of Unicode that assigns those codepoints. If
+ * mapping only assigned code points, the results of casefolding are
+ * guaranteed to be stable across Unicode versions (case conversion has weaker
+ * guarantees but still quite stable for assigned code points).
+ */
+
/*
* pg_strlower()
*
@@ -1326,6 +1386,8 @@ strupper_c(char *dst, size_t dstsize, const char *src, size_t srclen)
* Lowercasing is intended for human-readable display. If the goal is to
* convert to a canonical caseless form, see pg_strfold().
*
+ * See Case Mapping Complexities comment above.
+ *
* src must be in the database encoding with no embedded NULs. If dstsize is
* zero, dst may be NULL, which is useful for calculating the required buffer
* size before allocating.
@@ -1355,6 +1417,8 @@ pg_strlower(char *dst, size_t dstsize, const char *src, size_t srclen,
* titlecase form, if available), and all other characters lowercased. Used
* to implement the SQL INITCAP() function.
*
+ * See Case Mapping Complexities comment above.
+ *
* src must be in the database encoding with no embedded NULs. If dstsize is
* zero, dst may be NULL, which is useful for calculating the required buffer
* size before allocating.
@@ -1382,6 +1446,8 @@ pg_strtitle(char *dst, size_t dstsize, const char *src, size_t srclen,
* Uppercasing is intended for human-readable display. If the goal is to
* convert to a canonical caseless form, see pg_strfold().
*
+ * See Case Mapping Complexities comment above.
+ *
* src must be in the database encoding with no embedded NULs. If dstsize is
* zero, dst may be NULL, which is useful for calculating the required buffer
* size before allocating.
@@ -1415,6 +1481,8 @@ pg_strupper(char *dst, size_t dstsize, const char *src, size_t srclen,
* display. Unicode guarantees that casefolding is stable across versions if
* the original string consists only of assigned code points.
*
+ * See Case Mapping Complexities comment above.
+ *
* src must be in the database encoding with no embedded NULs. If dstsize is
* zero, dst may be NULL, which is useful for calculating the required buffer
* size before allocating.
diff --git a/src/test/regress/expected/collate.utf8.out b/src/test/regress/expected/collate.utf8.out
index cdd1a37ba18..3b8d6392732 100644
--- a/src/test/regress/expected/collate.utf8.out
+++ b/src/test/regress/expected/collate.utf8.out
@@ -194,7 +194,9 @@ INSERT INTO test_pg_unicode_fast VALUES
(U&'Λλ 1a \FF11a'),
('ȺȺȺ'),
('ⱥⱥⱥ'),
- ('ⱥȺ');
+ ('ⱥȺ'),
+ (U&'\FB01'),
+ (U&'\0390');
SELECT
t, lower(t), initcap(t), upper(t),
length(convert_to(t, 'UTF8')) AS t_bytes,
@@ -211,7 +213,9 @@ SELECT
ȺȺȺ | ⱥⱥⱥ | Ⱥⱥⱥ | ȺȺȺ | 6 | 9 | 8 | 6
ⱥⱥⱥ | ⱥⱥⱥ | Ⱥⱥⱥ | ȺȺȺ | 9 | 9 | 8 | 6
ⱥȺ | ⱥⱥ | Ⱥⱥ | ȺȺ | 5 | 6 | 5 | 4
-(7 rows)
+ fi | fi | Fi | FI | 3 | 3 | 2 | 2
+ ΐ | ΐ | Ϊ́ | Ϊ́ | 2 | 2 | 6 | 6
+(9 rows)
DROP TABLE test_pg_unicode_fast;
-- test Final_Sigma
diff --git a/src/test/regress/sql/collate.utf8.sql b/src/test/regress/sql/collate.utf8.sql
index 52cf068dd0c..6875f6f9e35 100644
--- a/src/test/regress/sql/collate.utf8.sql
+++ b/src/test/regress/sql/collate.utf8.sql
@@ -107,7 +107,9 @@ INSERT INTO test_pg_unicode_fast VALUES
(U&'Λλ 1a \FF11a'),
('ȺȺȺ'),
('ⱥⱥⱥ'),
- ('ⱥȺ');
+ ('ⱥȺ'),
+ (U&'\FB01'),
+ (U&'\0390');
SELECT
t, lower(t), initcap(t), upper(t),
--
2.43.0
