> On Sep 14, 2026, at 22:50, Andrew Dunstan <[email protected]> wrote: > > Hi, > > A linkedin post comparing CedarDB's new Unicode normalization support > to PostgreSQL's caught my eye [1]: same results, but a claimed 30x > speedup on "SELECT count(*) FROM hits WHERE url IS NORMALIZED" over > ClickBench's hits table. Most of that turned out to be down to CedarDB > using all available threads by default versus our > max_parallel_workers_per_gather of 2. But even at the matched thread > count they reported a 6x edge, attributed to two things: an ASCII fast > path (most URLs are already normalized ASCII, so you can skip decoding > entirely), and vectorized byte scanning for the ASCII check itself. > > I went and looked, and unicode_is_normalized(), unicode_assigned(), and > normalize() all decode every string to an array of char32_t codepoints, > one utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any > real work -- including on input that's already pure ASCII. The attached > patch adds a fast path: scan the raw bytes for anything with the high > bit set, using the SIMD-vectorized is_valid_ascii() we already have > (currently only used inside pg_utf8_verifystr()). If nothing is found, > the string is trivially normalized (ASCII code points have no > canonical or compatibility decomposition, and a combining class of > zero) and every code point in it is assigned, so all three functions > can return immediately. > > I deliberately didn't copy CedarDB's trick of comparing byte length to > codepoint count -- getting the codepoint count means calling > pg_mbstrlen_with_len(), exactly the scalar work this patch avoids. > Scanning raw bytes with is_valid_ascii() instead reuses SIMD > infrastructure we already have, and is cheaper to begin with: a single > reduction versus a population count. > > > Benchmarked with data sized to fit comfortably under shared_buffers rather > than triggering the seqscan ring-buffer bypass, which otherwise swamps the > comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15 > ASCII/non-ASCII mix, and no measurable regression on non-ASCII input > that still needs the full decode-and-quickcheck path. > > Regression tests cover the ASCII-hit case for all three functions, plus > a boundary sweep that plants a non-NFC sequence at varying offsets > around ASCII padding, to catch any off-by-one in the SIMD-chunk/scalar- > remainder split. > > > cheers > > > andrew > > > [1] https://lnkd.in/p/eKUqSj73 > > -- > Andrew Dunstan > EDB: https://www.enterprisedb.com > <0001-Add-ASCII-fast-path-to-Unicode-normalization-functio.patch>
The patch looks good to me. I also did some benchmark testing on my MacBook Air M4. I used clean builds with -O2 and without -g. # unicode_is_normalized() * all ascii: master 572ms; patch 68ms; Huge improvement * mixed: master 529ms; patch 64ms; Big improvement * non-ascii: master 399ms; patch 397ms; Roughly unchanged * late-non-ascii: master 1069ms; patch 1074ms; Roughly unchanged; This is the worse case, most of chars are ascii, and only unicode appear in the end # unicode_normalize_func() * all ascii: master 1659ms; patch 68ms; Huge improvement * mixed: master 1909ms; patch 233ms; Big improvement * non-ascii: master 1432ms; patch 1452ms; Roughly unchanged * late-non-ascii: master 3572ms; patch 3578ms; Roughly unchanged # unicode_assigned() * all ascii: master 258ms; patch 60ms; Big improvement * mixed: master 237ms; patch 61ms; Big improvement * non-ascii: master 221ms; patch 219ms; Roughly unchanged * late-non-ascii: master 531ms; patch 533ms; Roughly unchanged The late-non-ascii case is intended to be a worst case for the added ascii scan: most of the string is ascii, with the first non-ascii character appearing near the end. For pure ascii and mixed ascii/non-ascii input, all three functions show substantial improvements. For pure non-ascii input, including the late-non-ascii case, performance is roughly unchanged. So this looks like a worthwhile performance optimization to me. The attached is my test script. Best regards, -- Chao Li (Evan) HighGo Software Co., Ltd. https://www.highgo.com/
test_ascii_perf.sql
Description: Binary data
