ArnavBalyan commented on PR #8371: URL: https://github.com/apache/paimon/pull/8371#issuecomment-5158508170
> * Flink’s `SortedIndexTopoBuilder` uses signed bytes to sort the encoded keys, but `ReversedKeySerializer` uses unsigned bytes for comparison. > * Non-ASCII characters in UTF-8 result in reverse order, such as `a` and `é`. > * This may cause the `BTreeIndexWriter`’s monotonicity check to fail or generate a reverse B-tree with incorrect ordering. > * Existing tests only cover ASCII characters; it is recommended to add tests for non-ASCII keys and use a comparator consistent with the serializer. Hi @JingsongLi thanks a lot for the detailed review! I have added tests for the non ascii keys. I double checked the code path, looks like Flink bytes is resolving to SortUtil.compareBinary using unsigned comparator. I followed with the existing btree which uses unsigned. Have added the non ascii UT to ensure it also produces correct sort stream thanks! Let me know if you see any other concerns! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
