[
https://issues.apache.org/jira/browse/COLLECTIONS-899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109833#comment-18109833
]
尹茂椿萱 commented on COLLECTIONS-899:
----------------------------------
Thanks. The background of the report is actually a correctness analysis in my
research on property-based testing. I used a property derived from the
documented behavior of {{CaseInsensitiveMap}} and found that the actual
normalization differs from the behavior suggested by the documentation for some
Unicode characters, such as U+03D0 ({{{}ϐ{}}}).
So there is not a specific production use case behind the report. However, I
think the case still illustrates a potential source of confusion for users: if
they interpret “convert keys to lower case” as standard Unicode lowercase
conversion, they may not expect characters such as {{ϐ}} and {{β}} to be
normalized to the same key, which can result in an unexpected key collision.
This is why I think clarifying the Javadoc is useful even though the original
finding came from a correctness analysis rather than a concrete application.
And thanks for the additional information about case folding in Java 26. I
agree that implementing full Unicode case folding for hash calculation would be
a much larger change and is beyond what would be reasonable for this issue.
> CaseInsensitiveMap incorrectly converts U+03D0 (ϐ) to U+03B2 (β), causing
> distinct keys to collide
> --------------------------------------------------------------------------------------------------
>
> Key: COLLECTIONS-899
> URL: https://issues.apache.org/jira/browse/COLLECTIONS-899
> Project: Commons Collections
> Issue Type: Bug
> Components: Map
> Affects Versions: 4.6.0
> Environment: Apache Commons Collections 4.6.0
> Java: 17
> OS: Windows 11
> Reporter: 尹茂椿萱
> Priority: Major
> Labels: case-insensitive-map, regression,, unicode,
>
> {{CaseInsensitiveMap}} performs an incorrect Unicode lowercase conversion for
> some characters.
> The class-level Javadoc states that keys are converted to all lowercase in a
> locale-independent fashion using information from the Unicode data file.
> However, the current implementation converts each character using:
>
> {{Character.toLowerCase(Character.toUpperCase(chars[i]))}}
> This is not equivalent to Unicode lowercase conversion.
> For example, the Unicode character U+03D0 (GREEK BETA SYMBOL, {{{}ϐ{}}}) is
> unchanged by lowercase conversion:
>
> {{"\u03D0".toLowerCase(Locale.ROOT)}}
> returns {{{}ϐ{}}}.
> However, the current implementation converts it as follows:
>
> {{ϐ (U+03D0)
> -> Character.toUpperCase()
> β (U+03B2)
> -> Character.toLowerCase()
> β (U+03B2)}}
> As a result, {{ϐ}} and {{β}} are incorrectly normalized to the same key.
> This causes observable incorrect behavior in the public
> {{CaseInsensitiveMap}} API. For example:
>
> {{CaseInsensitiveMap<String, String> map = new CaseInsensitiveMap<>();
> map.put("\u03D0", "beta-symbol");
> map.put("\u03B2", "greek-beta");
> assertEquals(2, map.size());}}
> The expected size is {{{}2{}}}, because the two input keys are distinct and
> are not lowercase-equivalent according to Unicode lowercase mapping.
> With the current implementation, the actual size is {{1}} because both keys
> are converted to {{{}β{}}}.
> The previous implementation used:
>
> {{key.toString().toLowerCase()}}
> and preserved {{ϐ}} as {{{}ϐ{}}}, so this collision did not occur.
> This appears to be a regression introduced by the character-wise case
> conversion implementation.
> *Expected behavior*
> {{CaseInsensitiveMap}} should convert keys according to locale-independent
> Unicode lowercase semantics. In particular:
>
> {{U+03D0 (ϐ) -> U+03D0 (ϐ)
> U+03B2 (β) -> U+03B2 (β)}}
> Therefore, inserting both keys should retain two distinct entries.
> *Actual behavior*
> The current implementation converts:
>
> {{U+03D0 (ϐ) -> U+03B2 (β)
> U+03B2 (β) -> U+03B2 (β)}}
> Consequently, inserting both keys results in only one map entry.
> *Steps to reproduce*
> # Create a {{{}CaseInsensitiveMap{}}}.
> # Insert {{"\u03D0"}} as a key.
> # Insert {{"\u03B2"}} as a key.
> # Check the map size.
> Reproducer:
>
> {{CaseInsensitiveMap<String, String> map = new CaseInsensitiveMap<>();
> map.put("\u03D0", "beta-symbol");
> map.put("\u03B2", "greek-beta");
> System.out.println(map.size());}}
> Actual result:
>
> {{1}}
> Expected result:
>
> {{2}}
> *Possible fix*
> The issue appears to be caused by performing case conversion
> character-by-character using
> {{{}Character.toLowerCase(Character.toUpperCase(char)){}}}.
> A possible fix would be to use Java's string-level lowercase conversion with
> an explicit locale, for example:
>
> {{return key.toString().toLowerCase(Locale.ROOT);}}
> This preserves the intended locale-independent Unicode lowercase semantics
> described by the class-level documentation.
> If the character-wise implementation is retained for performance reasons, it
> should nevertheless ensure that its behavior is consistent with the
> documented Unicode lowercase mapping, including characters such as U+03D0
> ({{{}ϐ{}}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
