SteNicholas opened a new pull request, #485:
URL: https://github.com/apache/paimon-cpp/pull/485
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #405
<!-- What is the purpose of the change -->
A full-text search in Paimon C++ could not carry a row filter, so a caller
had to filter after
ranking and got fewer than `limit` rows. This adds
`FullTextSearchBuilder::WithFilter()`, aligned
with Java `FullTextSearchBuilder#withFilter` (apache/paimon#9855): only the
matching rows are
ranked, so the result is the top k among them, and they reach the existing
index as include row
ids, so BM25 statistics stay those of the full corpus.
- **Predicate split**: as Java
`PartitionPredicate#splitPartitionPredicatesAndDataPredicates` does,
conjuncts over partition fields alone are ANDed into the partition filter,
and the rest become the
data filter. A conjunct over both partition and data fields stays in the
data filter and also
restricts the partitions it implies, projected as Java
`PredicateProjectionConverter` does, so
`(p = 1 AND a > 0) OR (p = 2 AND a > 0)` also searches only `p = 1 OR p =
2`. Repeated calls are
ANDed.
- **`DataEvolutionFullTextScan`**: also keeps the index files that index a
filter field, and
attaches to each split the scalar pre-filter files that intersect its
shard: files that are not
full-text indexes, or full-text indexes with extra fields. Unlike in Java,
the latter rule holds
whichever column the index serves, as `lucene-fts` answers every scalar
predicate with all of its
rows. A scalar index file whose row range is negative, reversed or
overflows int64 fails the plan.
- **`IndexFullTextSearchSplit`**: version 2 of the encoding appends the
scalar index files, as
version 2 of Java `IndexFullTextSearchSplit` does; version 1 splits are
still read.
- **`DataEvolutionFullTextRead#matchedRows`**: resolves the data filter
before ranking with the
scalar indexes of the splits, read per field as one union
(`GlobalIndexScanImpl::
CreateForScalarFilters` and `ScanWithCoverage`, as Java
`DataEvolutionGlobalIndexScanner`).
- Rows that no scalar index covers are excluded, as in the `fast`
`scalar-index.search-mode`. If
no index can evaluate the filter at all, nothing is searched and a
warning is logged.
- If the indexes return only a superset, the candidates are excluded with
a warning by default,
so the search never returns a non-matching row. With the new
`global-index.filter.refine-from-data` option (default `false`, Java
description), they are
verified against the data instead.
- Each split searches with include row ids `searchRowRanges ∧ liveRows ∧
matchedRows`.
- **`GlobalIndexEvaluatorImpl`**: reports the fields that decided a result
(`EvaluateWithContributingFields`, as Java
`GlobalIndexEvaluator.Evaluation`), so that a superset
can be told from an exact answer: not every filter field contributed, or
there is a `Contains`,
`EndsWith` or `Like` leaf. Unlike in Java, a field with a conjunct that no
index could evaluate
does not count even if another of its conjuncts was evaluated, as the
result is then only a
superset for that field.
- **`FilteredRowIdReader`**: as in Java, reads only the filter columns and
`_ROW_ID` of the
candidates, pinned to the planned snapshot and with the schema that the
search resolved, so a
column renamed or replaced since then cannot point the filter at another
field. Unlike in Java,
once the planned snapshot has expired, a search of `scan.tag-name` reads
the tag instead, which
must still refer to the planned snapshot.
- **`StaticFromTagStartingScanner`**: reads tags from the branch of the
scan, as in Java, instead of
always from the main branch.
- **Options**: `scalar-index.search-mode` falls back on
`global-index.search-mode`, as Java
`CoreOptions#scalarIndexSearchMode` does. Only `fast` is supported so far;
a row filter in `full`
or `detail` fails with `NotImplemented`, as does a filter with a full-text
match.
Not part of this change:
- The `full` and `detail` scalar search modes (#406), which decide the rows
that no scalar index
covers by reading their filter columns.
- Row filters on a primary-key full-text index (#410), which is not
searchable yet.
### Tests
<!-- List UT and IT cases to verify this change -->
UT:
- `src/paimon/core/table/source/data_evolution_full_text_read_test.cpp`,
with fake scalar indexers:
`TestFilterRanksOnlyMatchedRows`,
`TestFilterExcludesRowsThatIndexesCannotDecide` (no scalar
file, a declining or multi-field one, a partly indexed conjunction,
`Contains`/`EndsWith`/`Like`,
rows that no scalar file covers),
`TestFilterUnionsIndexesOfDifferentTypes`, and filter validation in
`TestBuilderValidatesSettings` (unknown field, `_ROW_ID`, wrong type,
full-text match, a scalar
search mode other than `fast`, null filter).
- `src/paimon/core/table/source/data_evolution_full_text_scan_test.cpp`:
`TestScalarPreFilterFiles`
and `TestIntersectingFiles`, including invalid row ranges.
- `src/paimon/core/table/source/full_text_search_split_test.cpp`:
`TestSerializeRoundTrip` (the
version 2 bytes are pinned and version 1 bytes still read),
`TestSerializeScalarIndexFiles`, and
invalid scalar index files in `TestDeserializeRejectsInvalidBytes`,
`TestDeserializeRejectsCorruptEntries` and `TestCreateValidatesArguments`.
- `src/paimon/core/table/source/filtered_row_id_reader_test.cpp`:
`TestExactWhenEveryFilterFieldContributed` and
`TestCandidateOnlyLeavesAreNotExact`.
- `src/paimon/core/global_index/global_index_evaluator_impl_test.cpp`:
`ReportsContributingFields`,
`AndWithEmptyResultShortCircuits` and `CollectFieldIds`.
- `src/paimon/core/core_options_test.cpp`: `TestParseIndexSearchMode` for
the full-text and scalar
families, and `global-index.filter.refine-from-data` in `TestDefaultValue`
and `TestFromMap`.
IT (`test/inte/global_index_test.cpp`), aligned with Java
`NativeFullTextRowFilterTest` and
`FullTextSearchBuilderTest`:
- `TestFullTextTableFullTextSearchWithFilter` and
`TestLuceneTableFullTextSearchWithFilter`:
equality, range, `IN`, `NotEqual`, `OR` across fields, no and all matching
rows, equality on a
BTree of the text column, accumulated filters, the top row among matching
rows with `limit` 1,
a column without an index or with only a full-text index, a partly indexed
conjunction and
`Contains`/`EndsWith`/`Like` with refine on and off, scalar index files
attached only for filtered
columns, serialized splits, and refine with `scan.snapshot-id`,
`scan.tag-name` and
`scan.timestamp-millis`.
- `TestFullTextTableFullTextSearchWithPartitionAndDataFilter` and
`TestLuceneTableFullTextSearchWithPartitionAndDataFilter`: partition and
data predicates combined,
with `SetPartitionFilter()` and refine, and the partitions that a mixed
conjunct implies.
- `TestFullTextSearchSkipsRowsDeletedByDeletionVectors` and
`TestLuceneSearchSkipsRowsDeletedByDeletionVectors`: a filter with
deletion vectors, a scalar index
file per shard, and refine across shards.
- `TestFilteredRowIdReader`: deletion vectors, a pinned older snapshot, a
tag moved to another
snapshot, the tag fallback once the snapshot has expired (also with an
empty plan), a branch tag,
and a column renamed or replaced after planning.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
New public API: `FullTextSearchBuilder::WithFilter()`, and
`Options::SCALAR_INDEX_SEARCH_MODE` and
`Options::GLOBAL_INDEX_FILTER_REFINE_FROM_DATA`. The serialized form of
full-text search splits,
which is specific to Paimon C++, moves to version 2 and still reads version
1. No storage format or
protocol change.
### Documentation
<!-- Does this change introduce a new feature -->
Yes. `docs/source/user_guide/global_index.rst` adds a "Row Filters" section
covering the predicate
split, implied partitions, the `fast` mode, excluded candidates and
`global-index.filter.refine-from-data`, and the docs of
`FullTextSearchBuilder`, `FullTextScan`,
`FullTextRead` and the new options describe the filter.
### Generative AI tooling
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: Claude Code 2.1.311 (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
