JunRuiLee opened a new pull request, #752:
URL: https://github.com/apache/paimon-rust/pull/752
## What
Index files are always read from `<table>/index/<file>`. That ignores two
things Java records and
honors:
- `_EXTERNAL_PATH` — `IndexFileMeta.SCHEMA` field 5, `new DataField(5,
"_EXTERNAL_PATH",
newStringType(true))`. It is not decoded here at all, so it is also
dropped when this crate rewrites
an index manifest.
- `index-file-in-data-file-dir` — when set,
`FileStorePathFactory.indexFileFactory(partition, bucket)`
returns `IndexInDataFileDirPathFactory`, which resolves an index file
against the bucket's data-file
directory instead of the table `index/` directory.
A table with either of those fails to read its index files. Observed as a
primary-key vector search
against a table with `index-file-in-data-file-dir` enabled:
```
failed to open ANN index file
'.../<table>/index/index-<uuid>-0' for range reads
```
while the file is in the bucket directory.
## How
Decode `_EXTERNAL_PATH`, add it to the write schema so a rewritten manifest
keeps it, add the
`index-file-in-data-file-dir` option, and route every read-side index-file
consumer through one
resolver (`table/index_file_path.rs`) with two modes:
| mode | consumers | Java factory mirrored |
| --- | --- | --- |
| global — always `<table>/index` | data-evolution global index, vector
search, full-text search | `FileStorePathFactory.globalIndexFileFactory()`, as
used by `DataEvolutionGlobalIndexScanner` |
| bucket-local — data-file directory when the option is set | primary-key
vector ANN segments, primary-key full-text archives, deletion vectors |
`pathFactories.get(partition, bucket)`, as used by `IndexFileHandler.dvIndex` /
`pkVectorAnnSegment` |
An explicit external path wins over both layouts, matching
`globalIndexFileFactory().toPath(IndexFileMeta)`. The bucket directory is
taken from the split that
references the file rather than rebuilt from the table root, so custom data
directories and
postpone-bucket layouts keep working. The BTree reader cache is keyed by the
resolved path — with the
bare file name, two entries sharing a name but resolving to different
locations would reuse each
other's reader.
## Deliberately out of scope, called out for reviewers
Both of these would be a regression rather than a fix if only the read side
moved:
- **Dynamic-bucket hash index.** This crate writes it too, always to
`<table>/index`
(`prepare_commit_index` is a trait across five assigners taking a single
`index_dir`). Honoring the
layout on reads alone would stop it reading back what it just wrote. Read
and write have to move
together — a separate change.
- **`data-file.path-directory`.** Not supported here at all: there is no
core-option accessor, and
bucket paths are rooted directly at the table for data files as much as
for index files
(`DataFileMeta::data_file_path` consumes the same bucket path). Java
prepends the directory in
`FileStorePathFactory.relativeBucketPath`. Supporting it belongs with
data-file path handling, and
until then this change inherits the existing bucket path consistently
rather than diverging from it.
Write-side index builders are unchanged.
## Testing
New: resolver unit tests for both modes and external-path precedence;
deletion-vector tests for the
default layout, the data-file-directory layout, and external-path precedence
over both; a test that
`_EXTERNAL_PATH` survives `build_deletion_files_map`; decode tests for the
field present and absent.
Gates run locally from a clean HEAD: `cargo fmt --all -- --check`; `cargo
clippy -p paimon
--all-targets -- -D warnings` and the same with `--features fulltext`, both
clean; `cargo test -p
paimon --lib` 2417 passed / 0 failed and 2491 passed / 0 failed with
`--features fulltext`; integration
suites with no failures; `paimon-datafusion` and `paimon-c` build.
## Note
Opened as a draft. Stacked on #751 — the first commit on this branch is that
PR. There is no semantic
dependency; they share ~19 files of struct-literal sites, so they are
ordered rather than concurrent.
Review of this PR should look at the second commit only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
