jackylee-ch commented on PR #985: URL: https://github.com/apache/paimon-rust/pull/985#issuecomment-5945445362
Fixed by targeting the modern JVMs explicitly. `is_java_whitespace_only` reused the JDK 8 classification, which still counts U+180E as whitespace. Unicode 6.3 reclassified it as a format character, so on Java 9+ `Character.isWhitespace` returns false for U+180E and the value stays a literal partition — exactly the `dt=<U+180E>` directory your probe wrote. I dropped U+180E from the whitespace set so every partition-string path keeps it literal: the CHAR/VARCHAR native arm, the binary arm, and the Format Table `format_partition_value` all go through that one shared predicate, so they cannot diverge rather than extending the JDK 8 classification to each path. The compatibility target is now documented on the helper (modern JVMs, Java 11/17, Unicode >= 6.3). U+00A0 / U+2007 / U+202F stay unfolded and U+001C-U+001F stay folded, as before. Added your scenario as the SQL regression `test_u180e_partition_value_is_read_as_a_modern_jvm_literal` in `format_table_partition_filters.rs`: a real Parquet format table with a `dt=<U+180E>` directory, where `WHERE dt = '<U+180E>'` now returns the row. I verified non-vacuity end to end — restoring U+180E to the whitespace set makes that query return no rows and the test fails, and it also fails the added unit coverage on the string, binary, and datum formatting arms. The full `paimon` lib suite and the format-table integration tests pass; `clippy -p paimon --all-targets` and `-p paimon-datafusion --all-targets --features fulltext,vortex -D warnings` are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
