SteNicholas opened a new pull request, #461:
URL: https://github.com/apache/paimon-cpp/pull/461
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>):
<description> -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #404
<!-- What is the purpose of the change -->
Paimon C++ only exposed the reader primitive for full-text search: a caller
had to call
`GlobalIndexScan::CreateReaders`, run `VisitFullTextSearch` on the readers
and pass the result to
`ScanContextBuilder::SetGlobalIndexResult`, choosing index types and shard
ranges along the way,
with no serializable plan for running scan and read in different processes.
This adds the
table-level API, aligned with Java `FullTextSearchBuilder`,
`DataEvolutionFullTextScan` and
`DataEvolutionFullTextRead`.
- **`FullTextSearchBuilder`**: `SetQuery`, `SetLimit`, `SetPartitionFilter`,
`AddOption`/`SetOptions`,
`WithMemoryPool`/`WithExecutor`/`WithFileSystem`, `NewFullTextScan()`,
`NewFullTextRead()` and `ExecuteLocal()`. As in Java, the query column
must exist in the latest
schema and a read requires a positive limit. A column with a primary-key
full-text index
(`pk-full-text.index.columns`) is rejected with `NotImplemented` for now;
Java searches it with
`PrimaryKeyFullTextScan` and `PrimaryKeyFullTextRead`.
- **`DataEvolutionFullTextScan`**: resolves the snapshot as Java
`TimeTravelUtil#tryTravelOrLatest`
does, whatever `scan.mode` is: from `scan.snapshot-id`, `scan.tag-name`,
`scan.timestamp` or
`scan.timestamp-millis`, or else the latest one. It keeps the index files
that pass the partition
filter, index the column as their indexed field or as an extra field, and
whose indexer supports
full-text search. As Java `chooseIndexRanges` does (apache/paimon#8547),
it gives every row to
one shard across overlapping and multi-field indexes, a dedicated index
first, then by index
identity, start and end, and plans one split per chosen shard with its
physical range and its
assigned search row ranges. A `full-text` or `lucene-fts` index whose
backend is not built fails
the plan instead of silently dropping its rows.
- **`DataEvolutionFullTextRead`**: reads each split with an indexer created
from its own index
files, including their extra fields, through `OffsetGlobalIndexReader`,
ranking only its search
row ranges and live rows, and asks each split for `limit` candidates only
(apache/paimon#9955).
Splits are searched in parallel on the given executor, or on one with
`global-index.thread-num`
threads. The results are merged, keeping the first score of a duplicate
row id, and cut to the
global top-k by score descending, then row id ascending. Live rows are
pinned to the snapshot the
splits were planned from (apache/paimon#9031), which is loaded from a tag
once it has expired.
- **`FullTextSearchSplit` / `IndexFullTextSearchSplit`**: carry the snapshot
id, column, physical
range, search row ranges and index files, with a versioned encoding
specific to Paimon C++ (Java
uses Java serialization). Deserialization checks every count, range and
binary row layout before
decoding it, and the read rejects splits from different snapshots, of
another column, or whose
index files no longer index the field id of the column, such as after it
was replaced.
- **`GlobalIndexer::SupportsFullTextSearch()`**: `false` by default and
`true` for `full-text` and
`lucene-fts`, as Java `GlobalIndexerFactory#supportsFullTextSearch`.
- The scored top-k merge (`ScoredGlobalIndexResultUtils::MergeTopK`), the
row id range helpers
(`GlobalIndexUtils::RowIdsOf` / `RowIdsWithin`) and executor creation
(`GlobalIndexScanImpl::GetOrCreateExecutor`) are shared with the existing
union reader and scan.
Row filters (`withFilter`, #405) are not part of this change, and rows that
no index shard covers
are not searched.
### Tests
<!-- List UT and IT cases to verify this change -->
UT:
- `src/paimon/core/table/source/data_evolution_full_text_scan_test.cpp`:
`TestIndexIdentityAndIndexedFields`, `TestDisjointShards` (including a
shard ending at
`INT64_MAX`), `TestOverlappingShardsOfOneIndex`,
`TestDedicatedIndexBeforeExtraField`,
`TestIdentityOrderBreaksTies` and `TestServesFullTextSearch`.
- `src/paimon/core/table/source/data_evolution_full_text_read_test.cpp`,
with a fake full-text
indexer that records each search: `TestMergesShardResultsIntoGlobalTopK`,
`TestRanksOnlySearchRowRanges`, `TestIncludeRowIds`,
`TestReadsEachSplitWithItsOwnIndexFields`,
`TestRejectsInvalidSplits`, `TestRejectsSplitOfReplacedColumn`,
`TestBuilderValidatesSettings`,
`TestBuilderRejectsPrimaryKeyFullTextIndex` and `TestResolvesPlanSnapshot`.
- `src/paimon/core/table/source/full_text_search_split_test.cpp`:
`TestSerializeRoundTrip`,
`TestDeserializeRejectsInvalidBytes`,
`TestDeserializeRejectsCorruptFileLayout`,
`TestDeserializeRejectsCorruptEntries` and `TestCreateValidatesArguments`.
- `src/paimon/common/global_index/global_index_utils_test.cpp`:
`TestRowIdsOfAndWithin`.
- `SupportsFullTextSearch()` is checked in `global_indexer_factory_test.cpp`
(bitmap),
`full_text_global_index_test.cpp` and `lucene_global_index_test.cpp`.
IT (`test/inte/global_index_test.cpp`):
- `TestFullTextTableFullTextSearch` and `TestLuceneTableFullTextSearch`: two
shards, the table
search equals the reader search for every limit, serialized splits read
back, the snapshot
selected by `scan.snapshot-id` (also under `scan.mode` = `latest-full`),
`scan.tag-name` and
`scan.timestamp-millis`, a plan of an earlier snapshot, reading
`_INDEX_SCORE`, and a bitmap
index on the column is ignored.
- `TestFullTextSearchOverlappingIndexRanges` and
`TestLuceneSearchOverlappingIndexRanges`: the
planned shards and search ranges, and the table search equals the reader
search.
- `TestFullTextSearchSkipsRowsDeletedByDeletionVectors` and
`TestLuceneSearchSkipsRowsDeletedByDeletionVectors`: only live rows are
ranked, a plan made before
the deletion still ranks its rows, the read falls back to a tag once the
snapshot is deleted,
including when an unrelated tag is deleted while the tags are listed, and
fails when the snapshot
is missing or unreadable.
- `TestWriteCommitScanReadFullTextIndexWithPartition` and
`TestWriteCommitScanReadLuceneIndexWithPartition`: partition filters.
### API and Format
<!-- Does this change affect API in include dir or storage format or
protocol -->
New public API in `include/paimon/table/source/`:
`full_text_search_builder.h`, `full_text_scan.h`,
`full_text_read.h` and `full_text_search_split.h`, all marked
`PAIMON_EXPORT`. `GlobalIndexer`
gains the virtual `SupportsFullTextSearch()`, which defaults to `false`.
Full-text search splits
have a new serialized form (magic `PFTSPLT1`, version 1), specific to Paimon
C++. No storage format
or protocol change.
### Documentation
<!-- Does this change introduce a new feature -->
Yes. `docs/source/user_guide/global_index.rst` adds a "Table-Level Full-Text
Search" section,
covering local search, pinning the data scan to the plan snapshot and
distributed search with
serialized splits, and `docs/source/api/global_index.rst` lists the new
classes.
### Generative AI tooling
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
Generated-by: Claude Code 2.1.309 (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
