Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24559
to look at the new patch set (#2).
Change subject: IMPALA-12700: Take Iceberg Parquet Bloom filter properties into
account
......................................................................
IMPALA-12700: Take Iceberg Parquet Bloom filter properties into account
When writing Parquet files for Iceberg tables, Impala previously only
honored its own Bloom filter mechanisms: the 'parquet_bloom_filter_write'
query option and the 'parquet.bloom.filter.columns' table property. It
ignored the Iceberg-native Bloom filter write properties, so a table
configured for Bloom filters by another engine (Spark, Flink, PyIceberg)
got no Bloom filters when written by Impala.
This change makes Impala honor the Iceberg-native properties for Iceberg
tables:
- write.parquet.bloom-filter-enabled.column.<col>
- write.parquet.bloom-filter-max-bytes (default 1 MiB)
- write.parquet.bloom-filter-ndv.column.<col>
- write.parquet.bloom-filter-fpp.column.<col>
For each enabled column the Bloom filter bitset size is derived the same
way parquet-java sizes it: from the expected number of distinct values
(NDV) and the false positive probability (FPP), capped by
'bloom-filter-max-bytes'. The NDV is taken from the per-column property
if set, otherwise from the column's computed NDV statistic (COMPUTE
STATS) when available, otherwise the filter is sized at max-bytes
(matching parquet-java when the NDV is unknown). Using the column's
stats as a fallback is an improvement over parquet-java, which has no
access to such statistics.
The implementation is frontend-only: the backend Parquet writer already
consumes a column -> bitset-bytes map, so the size is computed in the
FE. The Iceberg-derived map is merged with the Impala-specific
'parquet.bloom.filter.columns' map; the Impala-specific property takes
precedence for any column present in both, keeping existing behavior
backward compatible.
The property values are validated at CREATE TABLE / ALTER TABLE SET
TBLPROPERTIES time (positive integer max-bytes and NDV, FPP in (0, 1)),
mirroring the existing Iceberg row-group-size / page-size validation.
Testing:
* New FE unit test IcebergBloomFilterUtilTest covering optimal sizing,
property validation, the NDV-from-stats fallback and column-name
handling.
* New negative cases in iceberg-negative.test for invalid property
values and setting the properties on a non-parquet Iceberg table.
* New e2e tests in test_parquet_bloom_filter.py that create Iceberg
tables with the properties, write data and verify the Bloom filters
Change-Id: Ic7baab79bbc055ddc8e2b05bbbc7083e02dc914e
Assisted-by: Claude Opus 4.8 (1M context) <[email protected]>
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M
fe/src/test/java/org/apache/impala/planner/ParquetBloomFilterTblPropParserTest.java
A fe/src/test/java/org/apache/impala/util/IcebergBloomFilterUtilTest.java
M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test
M tests/query_test/test_parquet_bloom_filter.py
10 files changed, 802 insertions(+), 76 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/24559/2
--
To view, visit http://gerrit.cloudera.org:8080/24559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic7baab79bbc055ddc8e2b05bbbc7083e02dc914e
Gerrit-Change-Number: 24559
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
