On 2026-09-15 Tu 12:22 AM, Chao Li wrote:
On Sep 14, 2026, at 22:50, Andrew Dunstan <[email protected]> wrote:
Hi,
A linkedin post comparing CedarDB's new Unicode normalization support
to PostgreSQL's caught my eye [1]: same results, but a claimed 30x
speedup on "SELECT count(*) FROM hits WHERE url IS NORMALIZED" over
ClickBench's hits table. Most of that turned out to be down to CedarDB
using all available threads by default versus our
max_parallel_workers_per_gather of 2. But even at the matched thread
count they reported a 6x edge, attributed to two things: an ASCII fast
path (most URLs are already normalized ASCII, so you can skip decoding
entirely), and vectorized byte scanning for the ASCII check itself.
I went and looked, and unicode_is_normalized(), unicode_assigned(), and
normalize() all decode every string to an array of char32_t codepoints,
one utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any
real work -- including on input that's already pure ASCII. The attached
patch adds a fast path: scan the raw bytes for anything with the high
bit set, using the SIMD-vectorized is_valid_ascii() we already have
(currently only used inside pg_utf8_verifystr()). If nothing is found,
the string is trivially normalized (ASCII code points have no
canonical or compatibility decomposition, and a combining class of
zero) and every code point in it is assigned, so all three functions
can return immediately.
I deliberately didn't copy CedarDB's trick of comparing byte length to
codepoint count -- getting the codepoint count means calling
pg_mbstrlen_with_len(), exactly the scalar work this patch avoids.
Scanning raw bytes with is_valid_ascii() instead reuses SIMD
infrastructure we already have, and is cheaper to begin with: a single
reduction versus a population count.
Benchmarked with data sized to fit comfortably under shared_buffers rather
than triggering the seqscan ring-buffer bypass, which otherwise swamps the
comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15
ASCII/non-ASCII mix, and no measurable regression on non-ASCII input
that still needs the full decode-and-quickcheck path.
Regression tests cover the ASCII-hit case for all three functions, plus
a boundary sweep that plants a non-NFC sequence at varying offsets
around ASCII padding, to catch any off-by-one in the SIMD-chunk/scalar-
remainder split.
[1] https://lnkd.in/p/eKUqSj73
The patch looks good to me.
I also did some benchmark testing on my MacBook Air M4. I used clean builds
with -O2 and without -g.
# unicode_is_normalized()
* all ascii: master 572ms; patch 68ms; Huge improvement
* mixed: master 529ms; patch 64ms; Big improvement
* non-ascii: master 399ms; patch 397ms; Roughly unchanged
* late-non-ascii: master 1069ms; patch 1074ms; Roughly unchanged; This is the
worse case, most of chars are ascii, and only unicode appear in the end
# unicode_normalize_func()
* all ascii: master 1659ms; patch 68ms; Huge improvement
* mixed: master 1909ms; patch 233ms; Big improvement
* non-ascii: master 1432ms; patch 1452ms; Roughly unchanged
* late-non-ascii: master 3572ms; patch 3578ms; Roughly unchanged
# unicode_assigned()
* all ascii: master 258ms; patch 60ms; Big improvement
* mixed: master 237ms; patch 61ms; Big improvement
* non-ascii: master 221ms; patch 219ms; Roughly unchanged
* late-non-ascii: master 531ms; patch 533ms; Roughly unchanged
The late-non-ascii case is intended to be a worst case for the added ascii
scan: most of the string is ascii, with the first non-ascii character appearing
near the end.
For pure ascii and mixed ascii/non-ascii input, all three functions show
substantial improvements. For pure non-ascii input, including the
late-non-ascii case, performance is roughly unchanged.
So this looks like a worthwhile performance optimization to me.
The attached is my test script.
Great, thanks for the review and tests.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
