839224346 opened a new pull request, #23:
URL: https://github.com/apache/paimon-full-text/pull/23
# [full-text] Enhance tokenizer with ICU segmentation, normalization, and
ngram filtering
## Summary
This PR adds three new capabilities to the native full-text index tokenizer
pipeline:
1. **Ngram `token_chars` filtering** — Unicode-aware character type
filtering for the ngram tokenizer using ICU `GeneralCategory`
2. **ICU normalize filter** — NFC/NFD/NFKC/NFKD Unicode normalization as a
token filter, with `is_normalized` fast-path optimization
3. **ICU word-boundary tokenizer** — A new `icu` tokenizer backed by
`icu_segmenter::WordSegmenter` for proper CJK and mixed-script segmentation
---
## New Configuration Options
On the Java side, `NativeFullTextGlobalIndexerFactory` calls
`options.removePrefix("full-text.")` to strip the prefix, then passes the
remaining key-value pairs through JNI to the Rust native layer. All options
below should be used with the `full-text.` prefix in table properties.
---
### 1. `full-text.tokenizer` (extended enum)
New value: `icu`.
| Value | Description | Use Case |
|---|---|---|
| `default` | Tantivy default tokenizer (Unicode word boundary +
configurable filters) | English and Latin-script languages |
| `simple` | Splits on non-letter characters, then lowercases | Simple
scenarios |
| `whitespace` | Splits only on whitespace | Pre-processed text |
| `raw` | No tokenization, entire field is a single token | Exact match,
keyword fields |
| `ngram` | Sliding window n-gram tokenization | Substring matching,
autocomplete |
| `jieba` | Jieba Chinese segmentation (dictionary-based) | Chinese text
requiring semantic word boundaries |
| **`icu`** (new) | ICU Unicode word boundary segmentation
(`WordSegmenter::new_auto()`) | Multilingual, mixed CJK/Latin text,
dictionary-free universal segmentation |
**Example:**
```sql
CREATE TABLE articles (
id INT,
content STRING,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'pk-full-text.index.columns' = 'content',
'full-text.tokenizer' = 'icu'
);
```
---
### 2. `full-text.ngram.token-chars` (new option)
| Property | Value |
|----------|-------|
| **Key** | `full-text.ngram.token-chars` |
| **Type** | Comma-separated list of character types |
| **Default** | empty (no filtering, all characters participate in ngrams) |
| **Applies when** | `full-text.tokenizer = ngram` |
#### Enum Values
| Value | ICU GeneralCategory Coverage | Example Characters |
|---|---|---|
| `letter` | UppercaseLetter, LowercaseLetter, TitlecaseLetter,
ModifierLetter, OtherLetter | `A` `z` `中` `の` `한` `α` |
| `digit` | DecimalNumber, LetterNumber, OtherNumber | `0`–`9` `①` `Ⅳ` |
| `whitespace` | SpaceSeparator, LineSeparator, ParagraphSeparator, and
Control chars where `is_whitespace()` is true | space, `\t`, `\n`, fullwidth
space |
| `punctuation` | ConnectorPunctuation, DashPunctuation, OpenPunctuation,
ClosePunctuation, InitialPunctuation, FinalPunctuation, OtherPunctuation | `.`
`,` `!` `(` `)` `—` `、` `。` |
| `symbol` | MathSymbol, CurrencySymbol, ModifierSymbol, OtherSymbol | `+`
`$` `€` `©` `°` |
#### Behavior
- Characters **not** in the specified types act as **token boundaries**
(they won't appear in generated ngrams)
- Multiple types are comma-separated, meaning "keep characters of these
types"
- Empty value (default) = all characters are kept, no filtering applied
#### Examples
```sql
-- Keep only letters and digits; punctuation acts as split boundary
'full-text.tokenizer' = 'ngram',
'full-text.ngram.min-gram' = '3',
'full-text.ngram.max-gram' = '3',
'full-text.ngram.token-chars' = 'letter,digit'
```
Input `"abc-123"` → splits into `"abc"` and `"123"` → ngrams: `["abc",
"123"]`
Input `"hello!"` → splits into `"hello"` → ngrams: `["hel", "ell", "llo"]`
Input `"中文测试"` → all are `letter`, kept → ngrams: `["中文测", "文测试"]`
```sql
-- Keep only letters (digits also become boundaries)
'full-text.ngram.token-chars' = 'letter'
```
Input `"abc123def"` → splits into `"abc"` and `"def"` → ngrams: `["abc",
"def"]`
---
### 3. `full-text.icu-normalize` (new option)
| Property | Value |
|----------|-------|
| **Key** | `full-text.icu-normalize` |
| **Type** | Normalization form enum |
| **Default** | empty (no Unicode normalization) |
| **Applies when** | Any tokenizer (cross-tokenizer filter) |
#### Enum Values
| Value | Full Name | Description | Typical Use |
|---|---|---|---|
| `nfc` | Canonical Decomposition + Canonical Composition | Composes
combining sequences into precomposed characters | General normalization, most
common default |
| `nfd` | Canonical Decomposition | Decomposes precomposed characters into
base + combining marks | When analyzing combining marks separately |
| `nfkc` | Compatibility Decomposition + Canonical Composition | NFC +
compatibility equivalence (fullwidth→halfwidth, ligatures→split, etc.) |
**Recommended**: multilingual search, CJK fullwidth/halfwidth unification |
| `nfkd` | Compatibility Decomposition | NFD + compatibility equivalence |
Same as NFKC but keeps decomposed form |
#### Normalization Effects Comparison
| Input | NFC | NFD | NFKC | NFKD |
|-------|-----|-----|------|------|
| `Ä` (U+00C4, precomposed) | `Ä` | `A` + `̈` (U+0308) | `Ä` | `A` + `̈` |
| `A` + `̈` (U+0041 U+0308) | `Ä` | `A` + `̈` | `Ä` | `A` + `̈` |
| `Ａ` (U+FF21, fullwidth A) | `Ａ` | `Ａ` | `A` | `A` |
| `fi` (U+FB01, ligature) | `fi` | `fi` | `fi` | `fi` |
| `①` (U+2460) | `①` | `①` | `1` | `1` |
| `㈱` (U+3231) | `㈱` | `㈱` | `株式会社` | `株式会社` |
#### Examples
```sql
-- NFKC: unify fullwidth/halfwidth, ideal for CJK mixed-text search
'full-text.icu-normalize' = 'nfkc'
```
Input `"Ａpache Ｐaimon"` → normalized to `"Apache Paimon"` → query `"apache"`
matches
```sql
-- NFC: unify combining character forms without compatibility mapping
'full-text.icu-normalize' = 'nfc'
```
Input `"naïve"` (decomposed `i` + `̈`) → composed to `"naïve"` (precomposed
`ï`) → consistent matching
```sql
-- Combined with ICU tokenizer
'full-text.tokenizer' = 'icu',
'full-text.icu-normalize' = 'nfkc',
'full-text.lower-case' = 'true'
```
#### Difference from `ascii-folding`
| Property | `icu-normalize` | `ascii-folding` |
|----------|----------------|-----------------|
| Implementation | ICU standard normalization | Hardcoded mapping table |
| Output | Still Unicode characters | Downgraded to ASCII only |
| `ü` → | `ü` (unchanged by NFC/NFKC) | `u` |
| `Ａ` → | `A` (NFKC) | `A` |
| Best for | Encoding form unification, fullwidth/halfwidth | Latin
diacritic removal for ASCII-only search |
| Composable | Yes, both can be enabled simultaneously | Yes |
---
## Full Configuration Examples
### Example 1: Chinese full-text search (ICU tokenizer + NFKC normalization)
```sql
CREATE TABLE docs (
id INT,
content STRING,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'pk-full-text.index.columns' = 'content',
'full-text.tokenizer' = 'icu',
'full-text.icu-normalize' = 'nfkc',
'full-text.lower-case' = 'true',
'full-text.remove-stop-words' = 'false'
);
```
### Example 2: Ngram substring matching (letters and digits only)
```sql
CREATE TABLE products (
id INT,
sku STRING,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'pk-full-text.index.columns' = 'sku',
'full-text.tokenizer' = 'ngram',
'full-text.ngram.min-gram' = '2',
'full-text.ngram.max-gram' = '4',
'full-text.ngram.token-chars' = 'letter,digit',
'full-text.lower-case' = 'true'
);
```
### Example 3: Per-column override (JSON)
```sql
CREATE TABLE multi_col (
id INT,
title STRING,
body STRING,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'pk-full-text.index.columns' = 'title',
'full-text.tokenizer' = 'default',
'fields.title.pk-full-text.index.options' =
'{"tokenizer":"icu","icu-normalize":"nfkc"}'
);
```
### Example 4: Python SDK
```python
options = {
"full-text.tokenizer": "icu",
"full-text.icu-normalize": "nfkc",
"full-text.lower-case": "true",
}
```
### Example 5: CALL statement
```sql
CALL sys.create_global_index(
table => 'db.articles',
index_column => 'content',
index_type => 'full-text',
options =>
'full-text.tokenizer=icu,full-text.icu-normalize=nfkc,full-text.lower-case=true'
);
```
---
## CJK Tokenizer Comparison
| Tokenizer | Pros | Cons | Use Case |
|-----------|------|------|----------|
| `jieba` | Dictionary-based, produces semantically meaningful Chinese words
| Chinese-only, adds binary size | Pure Chinese text needing semantic word
boundaries |
| `icu` | Language-agnostic, handles mixed scripts, no dictionary needed |
Character/short-word granularity for CJK (not semantic segmentation) |
Multilingual, mixed CJK/Latin, Korean, Japanese |
| `ngram` + `token-chars` | Guaranteed recall, works for any language |
Large index size, lower precision | Autocomplete, fuzzy substring matching |
---
## Testing
- 28 unit tests in `paimon-ftindex-core` covering all new features
- Full workspace: 71 tests pass, no clippy warnings
- New tests cover: `CharTypeMatcher` classification,
`FilteredNgramTokenizer`, ICU normalize (nfc/nfd/nfkc/nfkd), ICU tokenizer
(CJK, mixed scripts, punctuation handling, offsets)
## Commits
1. `367f15d` — `[full-text] Support ngram token_chars filtering`
2. `7cfea51` — `[full-text] Add ICU normalize filter`
3. `4807b3b` — `[full-text] Add ICU word-boundary tokenizer`
## Dependencies Added
- `icu_segmenter = "1.5"` — Word boundary detection (for ICU tokenizer)
- (existing) `icu_normalizer = "1.5"` — Unicode normalization (for ICU
normalize filter)
- (existing) `icu_properties = "1.5"` — `GeneralCategory` character
classification (for ngram token_chars)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
