QuakeWang commented on PR #454: URL: https://github.com/apache/paimon-rust/pull/454#issuecomment-4889501926
> I think there is a correctness issue for Binary/VarBinary BTree global indexes. > > `btree_global_index_build_builder.rs` currently allows `DataType::Binary(_)` and `DataType::VarBinary(_)`, but `btree/key_serde.rs` compares byte keys with Rust's default `a.cmp(b)`, i.e. unsigned lexicographic order. This is inconsistent with Paimon's predicate semantics: `spec/predicate.rs` compares `Datum::Bytes` with Java signed-byte order via `java_bytes_cmp`. > > For example, `0xff` and `0x00` have different ordering under these two comparators. If the BTree index is written and range-pruned with unsigned byte order while predicates use Java signed-byte order, range queries can miss matching rows. > > Could we either: > > 1. reject Binary/VarBinary for BTree global indexes for now, or > 2. make the BTree byte comparator use the same Java signed-byte ordering as predicate evaluation? > > A test with byte values like `[0xff]` and `[0x00]` would be useful to cover this. @JingsongLi Thanks. I fixed this conservatively by rejecting Binary/VarBinary for BTree global index build for now, and added a regression test for that. Supporting these types should be handled separately with aligned byte comparator semantics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
