singhpratech opened a new pull request, #51237:
URL: https://github.com/apache/arrow/pull/51237
### Rationale for this change
`utf8_normalize` with `form=NFC` or `NFKC` returned the decomposed forms
(NFD, NFKD). `Utf8NormalizeBase`
builds the right `utf8proc` options for each form but only calls
`utf8proc_decompose()`, which
decomposes regardless of `UTF8PROC_COMPOSE`; the composition step lives in
`utf8proc_normalize_utf32()`, which the kernel never called. See #51225.
### What changes are included in this PR?
- After a successful `utf8proc_decompose()`, when the options include
`UTF8PROC_COMPOSE`, call
`utf8proc_normalize_utf32()` on the scratch buffer. It composes in place
and returns the new code
point count; the existing UTF-8 encode loop is unchanged. NFD and NFKD
take the same path as before.
- Fix the `json_composed` fixture in `scalar_string_test.cc`: its bytes were
the decomposed form
(`61 CC 81`), the same string as `json_decomposed`, so the compose
assertions were comparing a value
with itself and passed with the bug. Thanks to @Santoshkumarpuppala for
spotting that on the issue.
- Add composed/decomposed pairs (U+00E9, and a Hangul syllable with its
jamo) to
`test_utf8_normalize` in pyarrow; the existing input, U+00B2, is its own
NFC.
### Are these changes tested?
Yes. With the corrected fixture, `TestStringKernels.Utf8Normalize` fails on
the unpatched kernel and
passes with this change; the pyarrow test covers the composed forms from
Python.
### Are there any user-facing changes?
Yes: `utf8_normalize` with `NFC` and `NFKC` now returns composed output.
Callers that depended on the
previous (decomposed) result for those forms will see different bytes.
**This PR contains a "Critical Fix".** (utf8_normalize produced wrong output
for two of its four forms.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
