On 18/10/2025 22:05, Collin Funk wrote:
Pádraig Brady <[email protected]> writes:
There were various other multi-byte blanks issues,
and multi-byte issues in general when I looked further.
The attached 3 further patches should make numfmt fully support multi-byte.
numfmt is a nice case where we don't need to optimize MB_CUR_MAX == 1,
thanks.
Right. But that got me thinking that we could optimize
in various cases, rather than resorting to mbsstr().
The attached implements mbsmbchr(mbs, mbc) to more efficiently
search for a multi-byte char in a multi-byte string,
especially with the usual UTF-8 charset
(which is determined with a single call to mbrtoc32() call per process).
cheers,
Padraig
From 50c8c41ab82a00b7a406936cd487323d0cb7faf2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= <[email protected]>
Date: Sun, 19 Oct 2025 13:11:46 +0100
Subject: [PATCH] numfmt: optimize multi-byte --delimiter
* src/numfmt.c (is_utf8_charset): A new function to efficiently
determine if running with a UTF-8 charset.
(mbsmbchr): A new function to efficiently search for
a (multi-byte) character in a multi-byte string.
(next-field): Use mbsmbchr() rather than mbstr() directly.
---
src/numfmt.c | 32 +++++++++++++++++++++++++++++++-
1 file changed, 31 insertions(+), 1 deletion(-)
diff --git a/src/numfmt.c b/src/numfmt.c
index 0f0a8770b..231d02d20 100644
--- a/src/numfmt.c
+++ b/src/numfmt.c
@@ -1363,6 +1363,36 @@ process_suffixed_number (char *text, long double *result,
return (e == SSE_OK || e == SSE_OK_PRECISION_LOSS);
}
+/* Return true if the current charset is UTF-8. */
+static bool
+is_utf8_charset (void)
+{
+ static int is_utf8 = -1;
+ if (is_utf8 == -1)
+ {
+ char32_t w;
+ mbstate_t mbstate = {0,};
+ is_utf8 = mbrtoc32 (&w, "\xe2\x9f\xb8", 3, &mbstate) == 3 && w == 0x27F8;
+ }
+ return is_utf8;
+}
+
+/* Search for multi-byte character C in multi-byte string S.
+ Return a pointer to the character, or nullptr if not found. */
+ATTRIBUTE_PURE
+static char *
+mbsmbchr (char const* s, char const* c)
+{
+ unsigned char uc = *c;
+ /* GB18030 is the most restrictive for the 0x30 optimization below. */
+ if (uc < 0x30 || MB_CUR_MAX == 1)
+ return strchr (s, uc);
+ else if (is_utf8_charset ())
+ return uc < 0x80 ? strchr (s, uc) : strstr (s, c);
+ else
+ return *(c + 1) == '\0' ? mbschr (s, *c) : (char *) mbsstr (s, c);
+}
+
/* Return a pointer to the beginning of the next field in line.
The line pointer is moved to the end of the next field. */
static char*
@@ -1373,7 +1403,7 @@ next_field (char **line)
if (delimiter)
{
- if (! *delimiter || ! (field_end = mbsstr (field_start, delimiter)))
+ if (! (field_end = mbsmbchr (field_start, delimiter)))
field_end = strchr (field_start, '\0');
}
else
--
2.51.0
