On Sun, Sep 29, 2019 at 3:38 AM Alvaro Herrera wrote:
>
> The UTF8 bits looks reasonable to me. I guess the other part of that
> question is whether we support any other multibyte encoding that
> supports combining characters. Maybe for cases other than UTF8 we can
> test for 0-width chars (using pg_encoding_dsplen() perhaps?) and drive
> the upper/lower decision off that? (For the UTF8 case, I don't know if
> Juanjo's proposal is better than pg_encoding_dsplen. Both seem to boil
> down to a bsearch, though unicode_norm.c's table seems much larger than
> wchar.c's).
>
Using pg_encoding_dsplen() looks like the way to go. The normalizarion
logic included in ucs_wcwidth() already does what is need to avoid the
issue, so there is no need to use unicode_norm_table.h. UTF8 is the
only multibyte encoding that can return a 0-width dsplen, so this
approach would also works for all the other encodings that do not use
combining characters.
Please find attached a patch with this approach.
Regards,
Juan José Santamaría Flecha
diff --git a/src/backend/utils/adt/formatting.c b/src/backend/utils/adt/formatting.c
index f7175df..8af62d2 100644
--- a/src/backend/utils/adt/formatting.c
+++ b/src/backend/utils/adt/formatting.c
@@ -1947,7 +1947,10 @@ str_initcap(const char *buff, size_t nbytes, Oid collid)
wchar_t*workspace;
size_t curr_char;
size_t result_size;
+int encoding;
+char wdsplen[MAX_MULTIBYTE_CHAR_LEN];
+encoding = GetDatabaseEncoding();
/* Overflow paranoia */
if ((nbytes + 1) > (INT_MAX / sizeof(wchar_t)))
ereport(ERROR,
@@ -1968,7 +1971,9 @@ str_initcap(const char *buff, size_t nbytes, Oid collid)
workspace[curr_char] = towlower_l(workspace[curr_char], mylocale->info.lt);
else
workspace[curr_char] = towupper_l(workspace[curr_char], mylocale->info.lt);
- wasalnum = iswalnum_l(workspace[curr_char], mylocale->info.lt);
+ wchar2char(wdsplen, [curr_char], MAX_MULTIBYTE_CHAR_LEN, mylocale);
+ if (pg_encoding_dsplen(encoding, wdsplen) != 0)
+ wasalnum = iswalnum_l(workspace[curr_char], mylocale->info.lt);
}
else
#endif
@@ -1977,7 +1982,9 @@ str_initcap(const char *buff, size_t nbytes, Oid collid)
workspace[curr_char] = towlower(workspace[curr_char]);
else
workspace[curr_char] = towupper(workspace[curr_char]);
- wasalnum = iswalnum(workspace[curr_char]);
+ wchar2char(wdsplen, [curr_char], MAX_MULTIBYTE_CHAR_LEN, mylocale);
+ if (pg_encoding_dsplen(encoding, wdsplen) != 0)
+ wasalnum = iswalnum(workspace[curr_char]);
}
}
