QuakeWang opened a new issue, #780: URL: https://github.com/apache/paimon-rust/issues/780
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Description Issue #34 was closed by PR #764 after landing the Java-compatible FileIndex outer format, Bitmap and Bloom Filter readers, built-in dispatch, and predicate/result composition. However, these components are currently used only by tests. The production `DataFileReader` path does not consume `DataFileMeta.embedded_index` or `.index` sidecars, so normal Rust reads cannot benefit from FileIndex pruning on Java-written tables. This task should integrate the existing FileIndex components into raw data-file reads, following Java Paimon's `FileIndexEvaluator` and `RawFileSplitRead`. #### Proposed scope - Support `file-index.read.enabled`, defaulting to `true` as in Java. - Resolve FileIndex data with Java-compatible precedence: - Prefer `embedded_index`. - Otherwise use one `.index` entry from `extra_files`, resolved with `aligned_file_path`. - Return `Remain` when no index exists. - Fail on multiple sidecars only when no embedded index is present. - Add a bytes-backed reader for embedded FileIndex data. - Evaluate predicates against the data file's schema: - Rebind fields by field ID to the file-level name, index, and type. - Safely devolve literals for schema evolution. - Fall back to `Remain` when predicate devolution is unsafe. - Skip unsupported index identifiers before reading their payloads, while continuing to use supported indexes from the same container. - Apply `FileIndexResult` before opening the data file: - `Remain` keeps the existing path. - `Skip` avoids reading the data file. - `Selection` is intersected with split row ranges and deletion vectors. - Keep the original data predicates as residual filters. FileIndex is pruning only and must not replace exact row-level filtering. - Use FileIndex selections only when the file row count fits the non-negative `i32` range used by the Java-compatible Bitmap format. - Preserve the existing conservative error behavior: structural format errors are propagated, while unsupported operations and lazy evaluation failures fall back to `Remain`. #### Tests Cover: - Embedded and sidecar indexes. - Regular and external-path sidecar resolution. - Bitmap selection and Bloom Filter false positives. - Intersection with deletion vectors and split row ranges. - Schema rename, reorder, added columns, and type promotion. - Unsupported identifiers and the `i32` row-count boundary. - Identical query results with FileIndex enabled and disabled. #### Out of scope - FileIndex writer and creation-option integration. - Additional index implementations, TopN, and limit-only evaluation. - Nested map-expression extraction. - `DataEvolutionReader` and merge-reader integration. - New public FileIndex APIs. This is a read-side follow-up to #34 and #764. Writer integration and additional reader or index types should be tracked separately. ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
