liwuhen opened a new pull request, #876:
URL: https://github.com/apache/paimon-rust/pull/876
### Purpose
Rust currently has two complementary forms of FileIndex coverage:
- Java golden payload tests that validate index encoding and predicate
evaluation.
- Rust write, commit, reload, and query tests that validate the internal Rust
read/write path.
However, there is no end-to-end test that opens a table written and committed
through Apache Paimon Java's normal table-writing path and then reads and
queries
that table through Rust.
Without this coverage, the combined compatibility of the FileIndex payload,
`DataFileMeta`, manifest metadata, and embedded or sidecar index paths is not
directly protected by a cross-language regression test.
This change adds reproducible Java-generated table fixtures and verifies that
Rust can load and use their FileIndexes through the normal scan and read
path.
The coverage includes:
- Bitmap and Bloom Filter indexes.
- Embedded and `.index` sidecar storage.
- Duplicate values, `NULL`, matching predicates, and non-matching predicates.
- FileIndex reads enabled and disabled.
- Row-level residual filtering after FileIndex evaluation.
- An observable data-file skip that cannot be attributed to manifest
statistics.
This is test coverage only. It does not indicate a known interoperability bug
and does not add new index types or writing modes.
### Brief change log
- Add four committed tables generated by Apache Paimon Java:
- `bitmap_embedded`
- `bitmap_sidecar`
- `bloom_filter_embedded`
- `bloom_filter_sidecar`
- Generate the fixtures through Java's normal catalog, append writer, commit,
snapshot, and manifest paths.
- Record the exact Apache Paimon Java commit, table options, input rows, and
expected query results.
- Retain the Java fixture generator and a regeneration script.
- Add a Rust integration test that:
- Loads the committed table schema and metadata.
- Plans and executes queries through the normal table read path.
- Compares results with `file-index.read.enabled=true` and `false`.
- Verifies duplicate and `NULL` values.
- Verifies row-level residual filtering.
- Checks the expected embedded or sidecar metadata representation.
- Uses the existing `FileIOProvider` observation pattern to prove that
FileIndex can skip opening the Parquet data file.
- Gate the filesystem-backed integration test on the `storage-fs` feature and
exclude Windows, consistently with existing filesystem fixture tests.
The fixtures were generated from Apache Paimon Java commit
`1d368b4a5932f8221fd28e2555001abdb8fb12ee` (`2.2-SNAPSHOT`).
The fixture rows, in physical order, are:
- `(1, "keep")`
- `(1, "drop")`
- `(NULL, "null-id")`
- `(3, "three")`
The observable skip predicate is `id = 2`. It lies inside the data file's
`[1, 3]` min/max range, so manifest statistics retain the file. With
FileIndex
reads enabled, Rust returns an empty result without opening the Parquet file.
With FileIndex reads disabled, Rust opens the Parquet file and row-level
filtering produces the same empty result.
### Tests
- Integration test `reads_java_written_file_indexes_and_prunes_data_files`
(`file_index_java_fixture_test.rs`):
- Covers Bitmap and Bloom Filter indexes in embedded and sidecar forms.
- Verifies that `id = 1` returns both duplicate rows.
- Verifies that `id IS NULL` returns the null row.
- Verifies that `id = 1 AND payload = 'keep'` retains row-level residual
filtering.
- Verifies that `id = 2` returns no rows with FileIndex reads both enabled
and
disabled.
- Verifies that manifest data statistics do not prune the `id = 2` query.
- Verifies that the data file is not opened when FileIndex reads are
enabled.
- Verifies that the data file is opened when FileIndex reads are disabled.
- The Java fixture generator completed successfully for all four tables.
- Commands run locally:
- `cargo test --locked -p paimon --test file_index_java_fixture_test`
- `cargo test --locked -p paimon --no-default-features --test
file_index_java_fixture_test --no-run`
- `cargo clippy --locked -p paimon --test file_index_java_fixture_test --
-D warnings`
- `cargo fmt --all -- --check`
- `bash -n crates/paimon/testdata/file_index/regenerate.sh`
- `git diff --check`
### API and Format
No public API or persisted data format changes.
This change only adds cross-language compatibility fixtures,
fixture-generation
support, and integration-test coverage. Production FileIndex readers,
writers,
metadata handling, and path resolution are not modified.
The scope does not include:
- Rust-written tables queried by Java.
- Range-bitmap writing.
- Primary-key or KV writing.
- Copy-on-write or data-evolution rewrites.
- Backfilling indexes for existing files.
- Performance benchmarks.
- New production metrics or public observability APIs.
### Documentation
Add fixture documentation describing:
- The exact Apache Paimon Java commit and version.
- The generator source and regeneration command.
- Bitmap and Bloom Filter configuration.
- Embedded and sidecar thresholds.
- Fixture rows and expected query results.
- The rationale for the observable FileIndex skip predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
