JingsongLi opened a new pull request, #24:
URL: https://github.com/apache/paimon-full-text/pull/24

   Adds `icu` and `icu/split` tokenizers backed by ICU4X's bundled dictionary 
data, following [Lance's ICU 
implementation](https://github.com/lance-format/lance/blob/main/rust/lance-tokenizer/src/icu.rs).
 Both handle Chinese/English mixed text without external model files. 
`icu/split` additionally splits punctuation-delimited identifiers such as 
`hello_world` and `alpha.beta` into searchable parts.
   
   Keeps existing Jieba support and filter defaults. Adds token and index 
round-trip tests for mixed scripts, UTF-8 offsets, positions, identifier 
boundaries, phrase search, English filters, and metadata persistence. Updates 
configuration documentation and regenerates the dependency inventory and native 
license reports.
   
   Validation:
   
   - `cargo test --workspace --locked`: 61 passed.
   - `cargo +1.90 test --workspace --locked`: 61 passed.
   - `cargo clippy --all-targets --workspace --locked -- -D warnings` and 
`cargo fmt --all -- --check`: passed.
   - Java/JNI tests: 2 passed; Python/FFI tests: 3 passed.
   - ASF license headers, dependency license check, dependency inventory, and 
`python3 tools/generate_license_reports.py --check`: passed.
   
   Known baseline failure: `cargo deny --locked check advisories` reports 
vulnerabilities in the existing `zstd` dependency chain (`zstd` 0.13.3, 
`zstd-safe` 7.2.4, and `zstd-sys` 2.0.16+zstd.1.5.7). These versions are 
unchanged from upstream `main` at `ee83ca7`; the ICU dependencies introduce no 
additional advisory failures.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to