[
https://issues.apache.org/jira/browse/IMPALA-12700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115777#comment-18115777
]
ASF subversion and git services commented on IMPALA-12700:
----------------------------------------------------------
Commit d569358c24a740adf11afa796d0da31fe184dc38 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d569358c2 ]
IMPALA-12700: Take Iceberg Parquet Bloom filter properties into account
When writing Parquet files for Iceberg tables, Impala previously only
honored its own Bloom filter mechanisms: the 'parquet_bloom_filter_write'
query option and the 'parquet.bloom.filter.columns' table property. It
ignored the Iceberg-native Bloom filter write properties, so a table
configured for Bloom filters by another engine (Spark, Flink, PyIceberg)
got no Bloom filters when written by Impala.
This change makes Impala honor the Iceberg-native properties for Iceberg
tables:
- write.parquet.bloom-filter-enabled.column.<col>
- write.parquet.bloom-filter-max-bytes (default 1 MiB)
- write.parquet.bloom-filter-ndv.column.<col>
- write.parquet.bloom-filter-fpp.column.<col>
For each enabled column the Bloom filter bitset size is derived the same
way parquet-java sizes it: from the expected number of distinct values
(NDV) and the false positive probability (FPP), capped by
'bloom-filter-max-bytes'. The NDV is taken from the per-column property
if set, otherwise from the column's computed NDV statistic (COMPUTE
STATS) when available, otherwise the filter is sized at max-bytes
(matching parquet-java when the NDV is unknown). Using the column's
stats as a fallback is an improvement over parquet-java, which has no
access to such statistics.
The implementation is frontend-only: the backend Parquet writer already
consumes a column -> bitset-bytes map, so the size is computed in the
FE. The Iceberg-derived map is merged with the Impala-specific
'parquet.bloom.filter.columns' map; the Impala-specific property takes
precedence for any column present in both, keeping existing behavior
backward compatible.
The property values are validated at CREATE TABLE / ALTER TABLE SET
TBLPROPERTIES time (a positive 32-bit max-bytes, a positive NDV, and an
FPP in (0, 1)), mirroring the existing Iceberg row-group-size / page-size
validation.
Another engine can set these properties without going through Impala's
DDL validation, so invalid values are also detected when Impala writes
the table. Such writes are not failed: the invalid properties are ignored
(no Bloom filters are produced, as before) and a warning is surfaced to
the user during analysis of the write. This covers every statement that
emits Iceberg data files - INSERT, UPDATE, MERGE and OPTIMIZE - which are
the paths where these properties would otherwise take effect.
Testing:
* New FE unit test IcebergBloomFilterUtilTest covering optimal sizing,
property validation, the NDV-from-stats fallback and column-name
handling.
* New FE analysis tests in AnalyzeDDLTest (TestIcebergParquetBloomFilter
Properties) for invalid property values and setting the properties on a
non-parquet Iceberg table, covering both CREATE TABLE and ALTER TABLE
SET TBLPROPERTIES.
* New e2e tests in test_parquet_bloom_filter.py that create Iceberg
tables with the properties, write data and verify the Bloom filters,
plus a test that injects an invalid property via Hive and asserts the
warning is surfaced (and the write still succeeds) for INSERT, UPDATE,
MERGE, and OPTIMIZE.
Change-Id: Ic7baab79bbc055ddc8e2b05bbbc7083e02dc914e
Assisted-by: Claude Opus 5 (Claude Code)
Reviewed-on: http://gerrit.cloudera.org:8080/24559
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Impala should take Iceberg bloom filter properties into consideration
> ---------------------------------------------------------------------
>
> Key: IMPALA-12700
> URL: https://issues.apache.org/jira/browse/IMPALA-12700
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Currenty Impala uses the query option "parquet_bloom_filter_write" and table
> property "parquet.bloom.filter.columns" to decide whether to write bloom
> filters or not.
> Iceberg uses different table properties:
> https://iceberg.apache.org/docs/1.4.0/configuration/
> * write.parquet.bloom-filter-enabled.column.col1
> * write.parquet.bloom-filter-max-bytes
> Impala should also respect the Iceberg table properties when it writes
> Parquet files in Iceberg tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
