Zoltán Borók-Nagy created IMPALA-15436:
------------------------------------------
Summary: [DOCS] Document the Iceberg write.parquet.bloom-filter-*
table properties
Key: IMPALA-15436
URL: https://issues.apache.org/jira/browse/IMPALA-15436
Project: IMPALA
Issue Type: Documentation
Reporter: Zoltán Borók-Nagy
Assignee: Zoltán Borók-Nagy
IMPALA-12700 makes Impala honor Iceberg's own Parquet Bloom filter properties
when writing Parquet files for Iceberg tables (INSERT, CTAS, UPDATE, MERGE,
OPTIMIZE). The "Iceberg table properties" list in
docs/topics/impala_iceberg.xml (concept iceberg_table_properties) ends at
write.parquet.dict-size-bytes. Add:
- write.parquet.bloom-filter-enabled.column.<col> — 'true' writes a Parquet
Bloom filter for that column (Parquet tables only)
- write.parquet.bloom-filter-max-bytes — positive 32-bit integer, Iceberg
default 1048576; an upper bound on the bitset, which Impala clamps to the 64 B
.. 128 MiB range
- write.parquet.bloom-filter-ndv.column.<col> — positive integer, the expected
number of distinct values used to size the filter. Without it Impala uses the
column's NDV statistic; with neither, the filter gets max-bytes.
- write.parquet.bloom-filter-fpp.column.<col> — false positive probability, 0 <
fpp < 1, default 0.01
Then explain:
- Impala validates the values in CREATE TABLE and ALTER TABLE SET
TBLPROPERTIES. CREATE TABLE also rejects these properties when
write.format.default is not PARQUET ("Parquet Bloom filter properties should be
set only for parquet file format"); ALTER checks only the values.
- If another engine stored invalid values, writes still succeed and warn:
"Ignoring invalid Iceberg Parquet Bloom filter properties: ...".
- The Impala-specific parquet.bloom.filter.columns property wins for a column
set in both.
- Writing is still gated by the PARQUET_BLOOM_FILTER_WRITE query option: NEVER
disables it, and the default IF_NO_DICT writes a filter only for row groups
that aren't fully dictionary-encoded.
A short example helps: CREATE TABLE ... STORED AS ICEBERG
TBLPROPERTIES('write.parquet.bloom-filter-enabled.column.id'='true','write.parquet.bloom-filter-ndv.column.id'='100000').
Pre-existing gap worth folding in, not caused by IMPALA-12700: Parquet Bloom
filters have no documentation at all. There is no page for
PARQUET_BLOOM_FILTER_WRITE or PARQUET_BLOOM_FILTERING, and
parquet.bloom.filter.columns ("col1:1024,col2", sizes rounded up to a power of
two within 64 B .. 128 MiB) is undocumented. Adding those pages,
with ditamap and keydef entries, gives the Iceberg properties something to link
to.
The behavior is additive, so no incompatible-changes entry.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
