Zoltan Borok-Nagy has uploaded this change for review. (
http://gerrit.cloudera.org:8080/24559
Change subject: WIP AI: IMPALA-12700: Take Iceberg Parquet Bloom filter
properties into account
......................................................................
WIP AI: IMPALA-12700: Take Iceberg Parquet Bloom filter properties into account
When writing Parquet files for Iceberg tables, Impala previously only
honored its own Bloom filter mechanisms: the 'parquet_bloom_filter_write'
query option and the 'parquet.bloom.filter.columns' table property. It
ignored the Iceberg-native Bloom filter write properties, so a table
configured for Bloom filters by another engine (Spark, Flink, PyIceberg)
got no Bloom filters when written by Impala.
This change makes Impala honor the Iceberg-native properties for Iceberg
tables:
- write.parquet.bloom-filter-enabled.column.<col>
- write.parquet.bloom-filter-max-bytes (default 1 MiB)
- write.parquet.bloom-filter-ndv.column.<col>
- write.parquet.bloom-filter-fpp.column.<col>
For each enabled column the Bloom filter bitset size is derived the same
way parquet-mr sizes it: from the expected number of distinct values (NDV)
and the false positive probability (FPP), capped by 'bloom-filter-max-bytes'.
The NDV is taken from the per-column property if set, otherwise from the
column's computed NDV statistic (COMPUTE STATS) when available, otherwise
the filter is sized at max-bytes (matching parquet-mr when the NDV is
unknown). Using the column's stats as a fallback is an improvement over
parquet-mr, which has no access to such statistics.
The implementation is frontend-only: the backend Parquet writer already
consumes a column -> bitset-bytes map, so the size is computed in the FE
and shipped via THdfsTableSink.parquet_bloom_filter_col_info. The
Iceberg-derived map is merged with the Impala-specific
'parquet.bloom.filter.columns' map; the Impala-specific property takes
precedence for any column present in both, keeping existing behavior fully
backward compatible.
The property values are validated at CREATE TABLE / ALTER TABLE SET
TBLPROPERTIES time (positive integer max-bytes and NDV, FPP in (0, 1)),
mirroring the existing Iceberg row-group-size / page-size validation.
Testing:
- New FE unit test IcebergBloomFilterUtilTest covering optimal sizing,
property validation, the NDV-from-stats fallback and column-name
handling.
- New negative cases in iceberg-negative.test for invalid property values
and setting the properties on a non-parquet Iceberg table.
- New end-to-end tests in test_parquet_bloom_filter.py that create Iceberg
tables with the properties, write data and verify the Bloom filters
written to the Parquet files (presence per column and bitset size),
including the stats fallback and the Impala-property precedence.
Change-Id: Ic7baab79bbc055ddc8e2b05bbbc7083e02dc914e
Assisted-by: Claude Opus 4.8 (1M context) <[email protected]>
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A fe/src/test/java/org/apache/impala/util/IcebergBloomFilterUtilTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
M tests/query_test/test_parquet_bloom_filter.py
8 files changed, 700 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24559/1
--
To view, visit http://gerrit.cloudera.org:8080/24559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic7baab79bbc055ddc8e2b05bbbc7083e02dc914e
Gerrit-Change-Number: 24559
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
