ktt proc <[email protected]> writes:
> I am very grateful for the software you have created.
Thanks for the kind words.
> I am sending you the error information regarding the coreutils sort program.
> It appears that it is unable to properly handle Unicode U+232a.
>
> I would appreciate it if you could review this.
[...]
> debian 13$ cut -f 1 ch | LANG=ko_KR.UTF8 sort -u
> 〈
> 〈
> 〉
> 「
> 」
> 『
> 』
>
> *"*U+232a*" missing in ko_KR.UTF8 environment*
This sort of stuff is left up to the strcoll function from libc in GNU
'sort'. That uses locale definitions which are up to the implementation.
They typically agree, but sometimes implementations make their own
choices.
I am not super familiar with glibc locales, but my understanding is that
their localedata/locales/ko_KR file does not list U+2329 and U+232A
under the LC_COLLATE section. As a result, they are both treated as
equal here.
I assume this was probably intentional since those characters are
deprecated. See some Unicode documentation suggesting alternatives [1]:
Deprecated angle brackets
These characters are deprecated and are strongly discouraged for
mathematical use because of their canonical equivalence to CJK
punctuation.
2329 〈 Left-Pointing Angle Bracket
→ 003C < less-than sign
→ 2039 ‹ single left-pointing angle quotation mark
→ 27E8 ⟨ mathematical left angle bracket
≡ 3008 〈 left angle bracket
232A 〉 Right-Pointing Angle Bracket
→ 003E > greater-than sign
→ 203A › single right-pointing angle quotation mark
→ 27E9 ⟩ mathematical right angle bracket
≡ 3009 〉 right angle bracket
You can email [email protected] if you believe the behavior
should change. Some people more familiar with locales than I should be
on that list.
Thanks,
Collin
[1] https://www.unicode.org/charts/nameslist/n_2300.html
