SteNicholas opened a new issue, #401: URL: https://github.com/apache/paimon-cpp/issues/401
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ### Motivation Sub-issue of #399 (step 2: query model). Java `org.apache.paimon.predicate.FullTextSearch` (apache/paimon#8308, apache/paimon#8459, apache/paimon#9855) is `(fieldName, query, limit)` plus a nullable `includeRowIds`: - `query` is a JSON DSL string. Java passes it unchanged to the native engine; the Java query parser `FullTextQuery` was removed in apache/paimon#8467. - `limit` is required and must be positive. - Results are always scored: a `ScoredGlobalIndexResult` with BM25 scores from the engine. - `includeRowIds` is a set of global row ids, held in a `RoaringNavigableMap64`. `offsetRange(from, to)` intersects it with the shard range `[from, to]` and converts it to shard-local ids. `include/paimon/predicate/full_text_search.h` in C++ takes a different model: - a `SearchType` enum (`MATCH_ALL`, `MATCH_ANY`, `PHRASE`, `PREFIX`, `WILDCARD`) - an optional `limit` - `with_score`, default `false` - `min_score` - `pre_filter` As a result, a query written for Java cannot run in C++. The `with_score` × `limit` matrix can also return unscored or unlimited results, which Java never produces. The JSON DSL is defined by apache/paimon-full-text `core/src/query.rs`: | Query | Fields | |---|---| | `match` | `query` (alias `terms`), optional `column`, `operator` (`Or` default / `And`), `boost` (1.0), `fuzziness` (0, a number or `"auto"`), `max_expansions` (50), `prefix_length` (0) | | `multi_match` | `query`, `columns`, `boosts`, `operator`, `fuzziness`, `max_expansions`, `prefix_length` | | `match_phrase` (alias `phrase`) | `query`, optional `column`, `slop` (0) | | `boolean` | `must`, `should`, `must_not` (and `queries` as `[occur, query]` pairs) | | `boost` | `positive`, `negative`, `negative_boost` (0.5) | Examples: `{"match":{"query":"paimon lake","operator":"And"}}`, `{"match_phrase":{"query":"paimon lake","slop":1}}`. ### Solution - Treat `FullTextSearch::query` as a JSON DSL string that the `full-text` backend passes through unchanged. Require `limit > 0` and always return scored results. An invalid query is reported by the engine as an error `Status`. - Replace `pre_filter` with an include-row-id set that has the Java `includeRowIds` semantics. Add an `OffsetRange(from, to)` helper for `OffsetGlobalIndexReader` (see {{S3}}). - If the scored-only model is adopted, change `GlobalIndexReader::VisitFullTextSearch` to return `ScoredGlobalIndexResult`. Update everything that implements or wraps it: - `UnionGlobalIndexReader` and `OffsetGlobalIndexReader` - the readers that reject full-text search: BTree, bitmap, lazy-filtered BTree, primary-key sorted and Lumina - Decide the fate of `SearchType`, `with_score` and `min_score`. The C++-only `lucene-fts` backend (`src/paimon/global_index/lucene/lucene_global_index_reader.cpp`) is built on `SearchType`. There are two options: 1. Keep them as a deprecated compatibility layer used only by `lucene-fts`, and have the `full-text` backend accept only the DSL form. 2. Remove them and translate the DSL for `lucene-fts`, rejecting DSL nodes it cannot express: - `match` → `TermQuery` / `BooleanQuery` with MUST or SHOULD - `match_phrase` → `PhraseQuery` `min_score` has no Java equivalent, and `PREFIX`/`WILDCARD` have no DSL equivalent. - Update the API documentation in `full_text_search.h` and the tests that build `FullTextSearch`. ### Anything else? This is a public API change under `include/paimon/`. Keep `PAIMON_EXPORT` on new symbols, and update callers, docs and tests together. Depends on {{S1}} for the `full-text` backend. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
