JunRuiLee opened a new pull request, #8101: URL: https://github.com/apache/paimon/pull/8101
### Purpose Implements the embedded bloom-filter file index read path from #8090 (PR 1 of the umbrella). On the PyPaimon read path, a data file whose embedded bloom index proves a queried value is absent is skipped before being read. This is a Python reimplementation of Java's file-index wire format and hashing, so it is byte/bit exact against the Java writer. **Changes** - New `fileindex` module: `BloomFilter64`/`BitSet`, `FastHash`, `FileIndexFormat` reader, `FileIndexPredicate` evaluator, `FileIndexResult`. - New option `file-index.read.enabled` (default `true`). - Pushdown integrated into `FileScanner._filter_manifest_entry`; explain gains a "File index pruning" funnel line. **Java compatibility** - `get_long_hash` reproduces Java's signed 64-bit long overflow; the bloom combined-hash reproduces signed 32-bit int overflow. - Strings/binary hash via XXH64 (matches `net.openhft LongHashFunction.xx`); FLOAT/DOUBLE via IEEE-754 bits; TINYINT/SMALLINT sign-truncate like Java's `(byte)`/`(short)`. - DATE/TIME/TIMESTAMP normalized to Java's internal repr (epoch-day / millis-of-day / millis-or-micros by precision). - `FileIndexFormat` reads start positions as signed ints, honors the `EMPTY_INDEX_FLAG` sentinel, and bounds-checks slices (a corrupt/truncated blob fails open instead of producing a short bitset). **Correctness gates** - **PK tables**: only applied in deletion-vector mode (files within a bucket don't overlap there), mirroring the existing value-stats gate, to avoid keeping a stale older value while skipping a newer overwrite. - **Data evolution**: skipped (Java devolves the filter by field id; out of scope here). - **Fail-open throughout**: unhashable literals and empty OR/IN degrade to REMAIN; any unexpected error keeps the file and is counted (`file_index_fail_open_count`, surfaced in explain). **Out of scope (tracked in #8090)** - External `.index` files (PR 4 in #8090) — only embedded indexes are read here. - CLI temporal literals (WHERE parser yields strings; safely fall back to REMAIN). - TIMESTAMP_LTZ / timezone-aware datetimes (rejected → fail open). - Modified-UTF-8 column names (decoded as standard UTF-8; ASCII in practice). ### Tests - **Cross-language e2e (authoritative, runs in CI `mixed_check`)**: `JavaPyE2ETest#testBloomFilterIndexWrite` writes embedded bloom indexes (BIGINT/INT/STRING) with the real Java writer and asserts an existing key is kept and an **in-range-but-absent** key is pruned; `JavaPyReadWriteTest::test_read_bloom_filter_index_table` reads them via PyPaimon and asserts the same. Absent keys are chosen inside each file's value-stats min/max so the prune decision falls on the bloom blob, not on value stats. Wired into `dev/run_mixed_tests.sh`. - **Unit** (plain `pytest`): `test_java_compat.py` pins each hash routine and the format-reader rules with Java-derived vectors, plus evaluator fail-open behavior (unhashable literal, empty OR/IN). - **Scanner integration**: `test_scanner_integration.py` builds a bloom blob with PyPaimon's own `BloomFilter64` and drives the scan path (prune / keep / disabled / explain counters). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
