On 2026-10-03 Sa 3:41 PM, Ayush Tiwari wrote:
Hi,
On Sat, 3 Oct 2026 at 19:57, Andrew Dunstan <[email protected]> wrote:
On 2026-09-18 Fr 10:11 AM, Andrew Dunstan wrote:
When time permits I'll send a new version, incorporating your review
and Bilal's suggestion.
Here's v3, rebased on current master. It folds in Bilal's change to pick
up where the ASCII scan stopped rather than decoding the whole string, and
deals with Tristan's review comments:
- The helper is now valid_ascii_prefix_len(), which works on raw bytes
rather than a text value, and returns the length of the valid ASCII
prefix it found, or the full length if the whole string is valid ASCII.
That gets rid of the -1 return in v2, and settles on "valid ASCII"
throughout, as Tristan suggested.
- The paragraph about Unicode normalization has moved out of the helper's
comment and into the callers, where it actually matters.
- As in Bilal's patch, normalize() and IS NORMALIZED restart with the last
ASCII character before the non-ASCII part, since a following combining
mark can compose with it. ASCII characters have no decomposition and
are all starters, so nothing earlier can be affected, and the quick
check gives the same answer starting there as from the beginning of the
string.
- There's a new regression test sweeping ASCII prefixes of 0 to 40 bytes
for normalize() (NFC and NFD) and unicode_assigned(), including a
combining mark right after the ASCII. It only lists lengths that give
the wrong answer, so the expected output is short. I checked that it
does fail if the last ASCII character isn't included in the part that
gets normalized.
I'll add it to the next CF.
I see CI tests failing for this patch. I haven't gone through the patch
yet, but from the logs it looks related to missing trailing spaces in
the newly added expected-output headings. The pg_upgrade failure seems
to be in its regression setup, not the upgrade itself.
Sorry about that. Here's a v4 that should have the whitespace issue fixed.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
From 27f411f766e2563e16426fa7a171cd6f1f07b298 Mon Sep 17 00:00:00 2001
From: Andrew Dunstan <[email protected]>
Date: Sun, 13 Sep 2026 17:13:15 -0400
Subject: [PATCH v4] Add ASCII fast path to Unicode normalization functions
unicode_is_normalized(), unicode_assigned(), and unicode_normalize_func()
each decoded its whole input to an array of char32_t codepoints, one
utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any real
work. ASCII characters need none of that: they have no canonical or
compatibility decomposition, a combining class of zero, and are all
assigned. So leading ASCII in a string is unchanged by normalization in
any form, except that its last character might compose with a following
combining mark.
Find the leading ASCII with a byte scan using the existing SIMD-vectorized
is_valid_ascii() (already used by pg_utf8_verifystr()). If the whole
string is ASCII, return at once. Otherwise decode and process only the
rest of the string, starting with the last ASCII character so that
composition with a following combining mark still happens.
Add regression coverage for the all-ASCII case in all three functions,
and sweeps over ASCII prefixes of varying length.
Author: Andrew Dunstan <[email protected]>
Co-Author: Nazir Bilal Yavuz <[email protected]>
Reviewed-by: Tristan Partin <[email protected]>
Reviewed-by: Chao Li <[email protected]>
Discussion: https://postgr.es/m/[email protected]
---
src/backend/utils/adt/varlena.c | 91 ++++++++++++++++++++++---
src/test/regress/expected/unicode.out | 96 +++++++++++++++++++++++++++
src/test/regress/sql/unicode.sql | 29 ++++++++
3 files changed, 206 insertions(+), 10 deletions(-)
diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c
index 6c95fcb24b7..11eb212384d 100644
--- a/src/backend/utils/adt/varlena.c
+++ b/src/backend/utils/adt/varlena.c
@@ -35,6 +35,7 @@
#include "parser/scansup.h"
#include "port/pg_bswap.h"
#include "regex/regex.h"
+#include "utils/ascii.h"
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/lsyscache.h"
@@ -167,6 +168,7 @@ static void text_format_string_conversion(StringInfo buf, char conversion,
int flags, int width);
static void text_format_append_string(StringInfo buf, const char *str,
int flags, int width);
+static int valid_ascii_prefix_len(const unsigned char *s, int len);
/*****************************************************************************
@@ -5475,6 +5477,33 @@ icu_unicode_version(PG_FUNCTION_ARGS)
PG_RETURN_NULL();
}
+/*
+ * Return the length of an initial portion of s[0..len) that is valid ASCII,
+ * that is, contains no zero bytes and no bytes with the high bit set, or
+ * len if all of it is.
+ *
+ * is_valid_ascii() requires a length that is a multiple of sizeof(Vector8),
+ * so check one chunk at a time with it and then the remainder byte by byte.
+ * A failure inside a chunk is reported at the start of that chunk, so the
+ * result can be up to sizeof(Vector8) - 1 bytes short of the full valid
+ * ASCII prefix.
+ */
+static int
+valid_ascii_prefix_len(const unsigned char *s, int len)
+{
+ int chunk_len = len - (len % sizeof(Vector8));
+
+ for (int i = 0; i < chunk_len; i += sizeof(Vector8))
+ if (!is_valid_ascii(s + i, sizeof(Vector8)))
+ return i;
+
+ for (int i = chunk_len; i < len; i++)
+ if (s[i] == 0 || IS_HIGHBIT_SET(s[i]))
+ return i;
+
+ return len;
+}
+
/*
* Check whether the string contains only assigned Unicode code
* points. Requires that the database encoding is UTF-8.
@@ -5484,15 +5513,23 @@ unicode_assigned(PG_FUNCTION_ARGS)
{
text *input = PG_GETARG_TEXT_PP(0);
unsigned char *p;
+ int len;
int size;
+ int start;
if (GetDatabaseEncoding() != PG_UTF8)
ereport(ERROR,
(errmsg("Unicode categorization can only be performed if server encoding is UTF8")));
+ /* ASCII code points are always assigned, so only check the rest */
+ len = VARSIZE_ANY_EXHDR(input);
+ start = valid_ascii_prefix_len((unsigned char *) VARDATA_ANY(input), len);
+ if (start == len)
+ PG_RETURN_BOOL(true);
+
/* convert to char32_t */
- size = pg_mbstrlen_with_len(VARDATA_ANY(input), VARSIZE_ANY_EXHDR(input));
- p = (unsigned char *) VARDATA_ANY(input);
+ size = pg_mbstrlen_with_len(VARDATA_ANY(input) + start, len - start);
+ p = (unsigned char *) VARDATA_ANY(input) + start;
for (int i = 0; i < size; i++)
{
char32_t uchar = utf8_to_unicode(p);
@@ -5519,26 +5556,42 @@ unicode_normalize_func(PG_FUNCTION_ARGS)
unsigned char *p;
text *result;
size_t i;
+ int len;
+ int start;
form = unicode_norm_form_from_string(formstr);
+ /*
+ * ASCII characters have no decomposition and a combining class of zero,
+ * so leading ASCII is unchanged by normalization in any form, except that
+ * its last character might compose with a following combining mark. So
+ * return an all-ASCII string as is, and otherwise normalize only the rest
+ * of the string, starting with that last ASCII character.
+ */
+ len = VARSIZE_ANY_EXHDR(input);
+ start = valid_ascii_prefix_len((unsigned char *) VARDATA_ANY(input), len);
+ if (start == len)
+ PG_RETURN_TEXT_P(input);
+ if (start > 0)
+ start--;
+
/* convert to char32_t */
- size = pg_mbstrlen_with_len(VARDATA_ANY(input), VARSIZE_ANY_EXHDR(input));
+ size = pg_mbstrlen_with_len(VARDATA_ANY(input) + start, len - start);
input_chars = palloc_array(char32_t, size + 1);
- p = (unsigned char *) VARDATA_ANY(input);
+ p = (unsigned char *) VARDATA_ANY(input) + start;
for (i = 0; i < size; i++)
{
input_chars[i] = utf8_to_unicode(p);
p += pg_utf_mblen(p);
}
input_chars[i] = (char32_t) '\0';
- Assert((char *) p == VARDATA_ANY(input) + VARSIZE_ANY_EXHDR(input));
+ Assert((char *) p == VARDATA_ANY(input) + len);
/* action */
output_chars = unicode_normalize(form, input_chars);
- /* convert back to UTF-8 string */
- size = 0;
+ /* convert back to UTF-8 string, after the unchanged ASCII */
+ size = start;
for (char32_t *wp = output_chars; *wp; wp++)
{
unsigned char buf[4];
@@ -5551,6 +5604,8 @@ unicode_normalize_func(PG_FUNCTION_ARGS)
SET_VARSIZE(result, size + VARHDRSZ);
p = (unsigned char *) VARDATA_ANY(result);
+ memcpy(p, VARDATA_ANY(input), start);
+ p += start;
for (char32_t *wp = output_chars; *wp; wp++)
{
unicode_to_utf8(*wp, p);
@@ -5587,20 +5642,36 @@ unicode_is_normalized(PG_FUNCTION_ARGS)
UnicodeNormalizationQC quickcheck;
size_t output_size;
bool result;
+ int len;
+ int start;
form = unicode_norm_form_from_string(formstr);
+ /*
+ * Leading ASCII is unchanged by normalization, except that its last
+ * character might compose with a following combining mark (see
+ * unicode_normalize_func()). So an all-ASCII string is normalized in
+ * every form, and otherwise only the rest of the string, starting with
+ * that last ASCII character, needs to be checked.
+ */
+ len = VARSIZE_ANY_EXHDR(input);
+ start = valid_ascii_prefix_len((unsigned char *) VARDATA_ANY(input), len);
+ if (start == len)
+ PG_RETURN_BOOL(true);
+ if (start > 0)
+ start--;
+
/* convert to char32_t */
- size = pg_mbstrlen_with_len(VARDATA_ANY(input), VARSIZE_ANY_EXHDR(input));
+ size = pg_mbstrlen_with_len(VARDATA_ANY(input) + start, len - start);
input_chars = palloc_array(char32_t, size + 1);
- p = (unsigned char *) VARDATA_ANY(input);
+ p = (unsigned char *) VARDATA_ANY(input) + start;
for (i = 0; i < size; i++)
{
input_chars[i] = utf8_to_unicode(p);
p += pg_utf_mblen(p);
}
input_chars[i] = (char32_t) '\0';
- Assert((char *) p == VARDATA_ANY(input) + VARSIZE_ANY_EXHDR(input));
+ Assert((char *) p == VARDATA_ANY(input) + len);
/* quick check (see UAX #15) */
quickcheck = unicode_is_normalized_quickcheck(form, input_chars);
diff --git a/src/test/regress/expected/unicode.out b/src/test/regress/expected/unicode.out
index 63e48d3a961..655dd99750a 100644
--- a/src/test/regress/expected/unicode.out
+++ b/src/test/regress/expected/unicode.out
@@ -26,6 +26,12 @@ SELECT unicode_assigned(U&'abc\+10FFFF');
f
(1 row)
+SELECT unicode_assigned('abc');
+ unicode_assigned
+------------------
+ t
+(1 row)
+
SELECT normalize('');
normalize
-----------
@@ -68,6 +74,19 @@ SELECT normalize(U&'\00E4\24D1c', NFKD) = U&'\0061\0308bc' COLLATE "C" AS test_n
t
(1 row)
+SELECT normalize('abc') = 'abc' COLLATE "C" AS test_ascii_idem;
+ test_ascii_idem
+-----------------
+ t
+(1 row)
+
+SELECT normalize(val) = val COLLATE "C" AS test_ascii_idem_column
+FROM (VALUES ('abc'::text)) v(val);
+ test_ascii_idem_column
+------------------------
+ t
+(1 row)
+
SELECT "normalize"('abc', 'def'); -- run-time error
ERROR: invalid normalization form: def
SELECT U&'\00E4\24D1c' IS NORMALIZED AS test_default;
@@ -82,6 +101,12 @@ SELECT U&'\00E4\24D1c' IS NFC NORMALIZED AS test_nfc;
t
(1 row)
+SELECT 'abc' IS NORMALIZED AS test_ascii;
+ test_ascii
+------------
+ t
+(1 row)
+
SELECT num, val,
val IS NFC NORMALIZED AS NFC,
val IS NFD NORMALIZED AS NFD,
@@ -105,6 +130,77 @@ ORDER BY num;
SELECT is_normalized('abc', 'def'); -- run-time error
ERROR: invalid normalization form: def
+-- Exercise the ASCII fast-path's chunk/remainder boundary handling: a
+-- non-NFC-normalized codepoint sequence must still be detected as such
+-- regardless of how much pure-ASCII padding surrounds it.
+SELECT len,
+ (repeat('a', len) || U&'\0061\0308') IS NORMALIZED AS non_nfc_at_end,
+ (U&'\0061\0308' || repeat('a', len)) IS NORMALIZED AS non_nfc_at_start
+FROM generate_series(0, 40) AS len
+ORDER BY len;
+ len | non_nfc_at_end | non_nfc_at_start
+-----+----------------+------------------
+ 0 | f | f
+ 1 | f | f
+ 2 | f | f
+ 3 | f | f
+ 4 | f | f
+ 5 | f | f
+ 6 | f | f
+ 7 | f | f
+ 8 | f | f
+ 9 | f | f
+ 10 | f | f
+ 11 | f | f
+ 12 | f | f
+ 13 | f | f
+ 14 | f | f
+ 15 | f | f
+ 16 | f | f
+ 17 | f | f
+ 18 | f | f
+ 19 | f | f
+ 20 | f | f
+ 21 | f | f
+ 22 | f | f
+ 23 | f | f
+ 24 | f | f
+ 25 | f | f
+ 26 | f | f
+ 27 | f | f
+ 28 | f | f
+ 29 | f | f
+ 30 | f | f
+ 31 | f | f
+ 32 | f | f
+ 33 | f | f
+ 34 | f | f
+ 35 | f | f
+ 36 | f | f
+ 37 | f | f
+ 38 | f | f
+ 39 | f | f
+ 40 | f | f
+(41 rows)
+
+-- normalize() and unicode_assigned() skip leading ASCII too, so check that
+-- they give the right answer whatever its length. A combining mark right
+-- after the ASCII must still compose with its last character. This lists
+-- only the lengths that give a wrong answer.
+SELECT len
+FROM generate_series(0, 40) AS len
+WHERE normalize(repeat('a', len) || U&'\0061\0308', NFC)
+ <> (repeat('a', len) || U&'\00E4') COLLATE "C"
+ OR normalize(repeat('a', len) || U&'\00E4' || repeat('b', len), NFD)
+ <> (repeat('a', len) || U&'\0061\0308' || repeat('b', len)) COLLATE "C"
+ OR normalize(repeat('a', len) || U&'\00E4' || repeat('b', len), NFC)
+ <> (repeat('a', len) || U&'\00E4' || repeat('b', len)) COLLATE "C"
+ OR unicode_assigned(repeat('a', len) || U&'\+10FFFF')
+ OR NOT unicode_assigned(repeat('a', len) || U&'\00E4' || repeat('b', len));
+ len
+-----
+(0 rows)
+
-- Hangul NFC recomposition tests
-- L+V -> LV composition (first and last)
SELECT normalize(U&'\1100\1161', NFC) = U&'\AC00' COLLATE "C" AS hangul_lv_first;
diff --git a/src/test/regress/sql/unicode.sql b/src/test/regress/sql/unicode.sql
index 951f86a336e..373745b32d3 100644
--- a/src/test/regress/sql/unicode.sql
+++ b/src/test/regress/sql/unicode.sql
@@ -8,6 +8,7 @@ SELECT U&'\0061\0308bc' <> U&'\00E4bc' COLLATE "C" AS sanity_check;
SELECT unicode_version() IS NOT NULL;
SELECT unicode_assigned(U&'abc');
SELECT unicode_assigned(U&'abc\+10FFFF');
+SELECT unicode_assigned('abc');
SELECT normalize('');
SELECT normalize(U&'\0061\0308\24D1c') = U&'\00E4\24D1c' COLLATE "C" AS test_default;
@@ -16,11 +17,15 @@ SELECT normalize(U&'\00E4bc', NFC) = U&'\00E4bc' COLLATE "C" AS test_nfc_idem;
SELECT normalize(U&'\00E4\24D1c', NFD) = U&'\0061\0308\24D1c' COLLATE "C" AS test_nfd;
SELECT normalize(U&'\0061\0308\24D1c', NFKC) = U&'\00E4bc' COLLATE "C" AS test_nfkc;
SELECT normalize(U&'\00E4\24D1c', NFKD) = U&'\0061\0308bc' COLLATE "C" AS test_nfkd;
+SELECT normalize('abc') = 'abc' COLLATE "C" AS test_ascii_idem;
+SELECT normalize(val) = val COLLATE "C" AS test_ascii_idem_column
+FROM (VALUES ('abc'::text)) v(val);
SELECT "normalize"('abc', 'def'); -- run-time error
SELECT U&'\00E4\24D1c' IS NORMALIZED AS test_default;
SELECT U&'\00E4\24D1c' IS NFC NORMALIZED AS test_nfc;
+SELECT 'abc' IS NORMALIZED AS test_ascii;
SELECT num, val,
val IS NFC NORMALIZED AS NFC,
@@ -37,6 +42,30 @@ ORDER BY num;
SELECT is_normalized('abc', 'def'); -- run-time error
+-- Exercise the ASCII fast-path's chunk/remainder boundary handling: a
+-- non-NFC-normalized codepoint sequence must still be detected as such
+-- regardless of how much pure-ASCII padding surrounds it.
+SELECT len,
+ (repeat('a', len) || U&'\0061\0308') IS NORMALIZED AS non_nfc_at_end,
+ (U&'\0061\0308' || repeat('a', len)) IS NORMALIZED AS non_nfc_at_start
+FROM generate_series(0, 40) AS len
+ORDER BY len;
+
+-- normalize() and unicode_assigned() skip leading ASCII too, so check that
+-- they give the right answer whatever its length. A combining mark right
+-- after the ASCII must still compose with its last character. This lists
+-- only the lengths that give a wrong answer.
+SELECT len
+FROM generate_series(0, 40) AS len
+WHERE normalize(repeat('a', len) || U&'\0061\0308', NFC)
+ <> (repeat('a', len) || U&'\00E4') COLLATE "C"
+ OR normalize(repeat('a', len) || U&'\00E4' || repeat('b', len), NFD)
+ <> (repeat('a', len) || U&'\0061\0308' || repeat('b', len)) COLLATE "C"
+ OR normalize(repeat('a', len) || U&'\00E4' || repeat('b', len), NFC)
+ <> (repeat('a', len) || U&'\00E4' || repeat('b', len)) COLLATE "C"
+ OR unicode_assigned(repeat('a', len) || U&'\+10FFFF')
+ OR NOT unicode_assigned(repeat('a', len) || U&'\00E4' || repeat('b', len));
+
-- Hangul NFC recomposition tests
-- L+V -> LV composition (first and last)
SELECT normalize(U&'\1100\1161', NFC) = U&'\AC00' COLLATE "C" AS hangul_lv_first;
--
2.43.0
