On 2025-11-20 Th 7:55 AM, Nazir Bilal Yavuz wrote:
Hi,
Thank you for looking into this!
On Thu, 20 Nov 2025 at 00:01, Nathan Bossart <[email protected]> wrote:
IMHO we should be looking for ways to simplify this should-we-use-SIMD
code. For example, perhaps we could just disable the SIMD path for 10K or
100K lines any time a special character is found. I'm dubious that a lot
of complexity is warranted.
I think this is a bit too harsh since SIMD is still worth it if SIMD
can advance more than ~5 character average. I am trying to use SIMD as
much as possible when it is worth it but what you said can remove the
regression completely, perhaps that is the correct way.
Perhaps a very small regression (say under 1%) in the worst case would
be OK. But the closer you can get that to zero the more acceptable this
will be. Very large loads of sparse data, which will often have lots of
special characters AIUI, are very common, so we should not dismiss the
worst case as an outlier. I still like the idea of testing, say, a
thousand lines every million, or something like that.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
