wombatu-kun opened a new pull request, #8839: URL: https://github.com/apache/paimon/pull/8839
### Purpose Resolves the TODO in `DataEvolutionSplitRead.withFilter`, which was a no-op: Support File index push down (all conditions) and Predicate push down (only if no column merge). As a result, data evolution tables never used their file indexes on read: `AppendOnlyFileStoreScan` evaluates the embedded index in `filterByStats(ManifestEntry)`, but `DataEvolutionFileStoreScan` overrides that method and only checks row id ranges. Bloom, bitmap and bsi indexes were written for data evolution files and never read back. Format level push down did not reach the reader either, since the `FormatReaderMapping.Builder` was built with `filters = null`. Push down is now applied wherever it can not interfere with the column merging: - **No column merge**: the file index can skip the file, a `BitmapIndexResult` narrows the read to the matching positions, and the filters reach the format reader, mirroring `RawFileSplitRead`. - **Merged group**: whole group skipping only. `DataEvolutionFileReader` zips its inner readers positionally, so dropping rows in one of them would break the column alignment. Skipping the group is sound because a logical row takes each column from exactly one file. The existing `enabledFilterPushDown` flag of `FormatReaderMapping.Builder` already draws that line (the merge path calls `build(..., false)`), so passing `filters` into the builder needs no extra branching. Two safety points: - Only plain data files may skip a merged group. `mergeRangesAndSort` asserts `areAllRangesSame(dataFiles)`, while a blob or vector-store file covers only a sub range and proves nothing about the other rows. - Filters on `_ROW_ID` and `_SEQUENCE_NUMBER` are never pushed down: they are assigned from the manifest entry, and data evolution may reassign row ids, so a physical copy in the file can be stale. Semantics are unchanged: `withFilter` is a hint, and this only drops rows proven not to match. Benchmark (throwaway, not in the diff): 200k rows in 40 row id groups, ids interleaved so min/max stats prune nothing, bloom filter, point filter. 39 of 40 groups skipped. | Scenario | Rows read | best | Speedup | | --- | --- | --- | --- | | Groups of one file | 200,000 to 5,000 | 74-79 ms to 21-24 ms | ~3.4x | | Merged groups | 200,000 to 5,000 | 83-90 ms to 23-24 ms | ~3.7x | Left for a follow up: row level bitmap selection inside a merged group, which needs the bitmap translated into global row id ranges and fed through the existing `rowRanges` mechanism. ### Tests New `DataEvolutionFileIndexTest` (7 tests). Readers are created directly from the splits without `executeFilter()`, so an empty result proves the reader itself pruned the rows; filter values sit inside the column min/max range so the scan does not drop the split first. Covered: a single file group skipped by a standalone `.index` file, a merged group skipped by an embedded index, column alignment preserved in a merged group, exact bitmap index selection, a merged group skipped after `RENAME COLUMN` (the filter has to be devolved by field id), `file-index.read.enabled = false`, and a filter mixing `_ROW_ID` with a data column. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
