This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8f3432e16a13 docs(performance): document Parquet bloom filters and
separate them from the bloom index (#19590)
8f3432e16a13 is described below
commit 8f3432e16a13e1587272843ce617973e6d44bbb4
Author: Ranga Reddy <[email protected]>
AuthorDate: Sun Aug 30 20:04:56 2026 +0530
docs(performance): document Parquet bloom filters and separate them from
the bloom index (#19590)
Parquet's per-column bloom filters have been passthrough-configurable in
Hudi since 0.14.0, but the
only trace of them in the docs was an auto-generated row in
configurations.md - and that row is for
a different feature (see below). Nothing told a user the keys exist, that
they are per column, or
that they are read from the Hadoop config.
Adds a "Parquet Bloom Filters" subsection to performance.md under Read
Path, next to Data Skipping,
since that is the gap it fills: column stats prune on ranges and are least
useful for an equality
predicate on a high-cardinality column, which is exactly what a bloom
filter covers.
parquet.bloom.filter.enabled#<column> write a bloom filter for that
column
parquet.bloom.filter.expected.ndv#<column> expected distinct values,
sizes the filter
HoodieBaseParquetWriter.handleParquetBloomFilters scans the Hadoop
configuration for those prefixes
and forwards each to the Parquet writer builder, so from Spark they are set
with the spark.hadoop.
prefix. The spelling matches TestHoodieParquetBloom, which sets exactly
these keys on
jsc.hadoopConfiguration.
Two things are called out because both are easy to get wrong:
- hoodie.parquet.bloom.filter.enabled is NOT this feature, despite the
name. It controls whether
Hudi writes a bloom filter of RECORD KEYS into the footer for the bloom
index during upserts; it
defaults to true, applies only when meta fields are populated, and is
implied when
hoodie.index.type names a BLOOM index
(HoodieFileWriterFactory.enableBloomFilter). Setting it
does not enable per-column Parquet filters.
- the settings are applied by reflection and the NoSuchMethodException is
swallowed, so on a
Parquet without withBloomFilterEnabled / withBloomFilterNDV they are
silently ignored.
Applied to the current docs and to every supported versioned copy - 1.2.0,
1.1.1, 1.0.2, 1.0.1 and
1.0.0 - because the feature ships in all of them: handleParquetBloomFilters
is absent at
release-0.13.1 and present from release-0.14.0 onwards, and the
hoodie.parquet.bloom.filter.enabled
config it is contrasted against reads
defaultValue(true)/sinceVersion(0.15.0) identically at
release-1.0.0, 1.0.2, 1.1.1 and 1.2.0, so the text is accurate for each.
That five-version fan-out
matches how other cross-version docs fixes have been applied (#19555,
#19459). version-1.0.0 has no
"Related Resources" heading, so there the section is appended at the end of
the page instead of
before it; the inserted text is byte-identical in all six files.
The 0.15.x and 0.14.x copies are left alone: they have had no or nearly no
doc commits in the last
six months and are effectively frozen.
Review follow-up: says what the read side needs, since the section sits
under Read Path while the
keys it documents are write-side. On Spark 3.x nothing extra is configured
- reading back through the
Hudi datasource consults the filters given a pushed-down equality
predicate, and
TestHoodieParquetBloomFilter pins that by disabling the statistics and
column-index filter levels so
only a bloom can cause a skip, then asserting zero row groups scanned for
an absent value against one
for a present value. parquet.filter.bloom.enabled is left at its parquet-mr
default and Hudi never
sets it. Also renamed the example column from uuid, which read like a
record key, and said outright
that <column> is any data column filtered by equality rather than the
record key.
Closes #16063
---
website/docs/performance.md | 50 ++++++++++++++++++++++
.../versioned_docs/version-1.0.0/performance.md | 50 ++++++++++++++++++++++
.../versioned_docs/version-1.0.1/performance.md | 50 ++++++++++++++++++++++
.../versioned_docs/version-1.0.2/performance.md | 50 ++++++++++++++++++++++
.../versioned_docs/version-1.1.1/performance.md | 50 ++++++++++++++++++++++
.../versioned_docs/version-1.2.0/performance.md | 50 ++++++++++++++++++++++
6 files changed, 300 insertions(+)
diff --git a/website/docs/performance.md b/website/docs/performance.md
index 7c6d489c3732..e540e770fb75 100644
--- a/website/docs/performance.md
+++ b/website/docs/performance.md
@@ -132,6 +132,56 @@ To enable Data Skipping in your queries make sure to set
following properties to
- `hoodie.metadata.enable` (to enable metadata table use on the read path,
enabled by default)
- `hoodie.metadata.index.column.stats.enable` (to enable column stats index
use on the read path)
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most:
an equality predicate on a
+high-cardinality column whose min-max range covers almost every file.
Parquet's own bloom filters cover that
+case. They are written into the Parquet file itself, and a reader consults
them to skip row groups that
+cannot contain the value being searched for.
+
+Hudi passes these through to the Parquet writer, per column, from the
**Hadoop** configuration:
+
+| key | meaning |
+| --- | --- |
+| `parquet.bloom.filter.enabled#<column>` | write a bloom filter for
`<column>` |
+| `parquet.bloom.filter.expected.ndv#<column>` | expected number of distinct
values, which sizes the filter |
+
+`<column>` is any data column you filter on by equality — not the record key,
and unrelated to the record-key
+bloom index discussed in the note below. Set the keys on the Hadoop
configuration your writer uses; from Spark
+the `spark.hadoop.` prefix forwards them:
+
+```
+--conf spark.hadoop.parquet.bloom.filter.enabled#session_id=true
+--conf spark.hadoop.parquet.bloom.filter.expected.ndv#session_id=100000
+```
+
+Give `expected.ndv` a realistic estimate for the column. Too low and the
filter saturates and stops
+eliminating anything; too high and you pay in file size for nothing.
+
+This is a write-time decision: only files written after you set it carry the
filters, so an existing table
+picks them up as it is rewritten by ongoing writes, compaction or clustering.
+
+On the read side nothing extra needs configuring for Spark 3.x. Reading the
table back through the Hudi
+datasource consults the filters, provided the query carries an equality
predicate that can be pushed down to
+the Parquet reader — a query filtering on a value no row group contains skips
those row groups entirely.
+Parquet's own read-side switch, `parquet.filter.bloom.enabled`, is left at its
default and Hudi never
+overrides it, so there is no reader-side flag to turn on.
+
+:::note
+Do not confuse the keys above with Hudi's own
`hoodie.parquet.bloom.filter.enabled`, which is a different
+feature despite the near-identical name. That config controls whether Hudi
writes a bloom filter **of record
+keys** into the file footer for use by the [bloom index](indexes.md) during
upserts; it defaults to `true`,
+applies only when meta fields are populated, and is implied anyway when
`hoodie.index.type` names a `BLOOM`
+index. It has nothing to do with per-column skipping on the read path, and
setting it does not enable the
+Parquet column filters described here.
+:::
+
+:::caution
+Hudi applies these settings reflectively, so if the Parquet version on your
classpath predates the
+`withBloomFilterEnabled` / `withBloomFilterNDV` builder methods, the keys are
**silently ignored** rather
+than rejected. If you see no change in file size or query behaviour, check
your Parquet version first.
+:::
+
## Related Resources
<h3>Blogs</h3>
diff --git a/website/versioned_docs/version-1.0.0/performance.md
b/website/versioned_docs/version-1.0.0/performance.md
index fece8c260b18..1840d0f65ab9 100644
--- a/website/versioned_docs/version-1.0.0/performance.md
+++ b/website/versioned_docs/version-1.0.0/performance.md
@@ -131,3 +131,53 @@ To enable Data Skipping in your queries make sure to set
following properties to
- `hoodie.enable.data.skipping` (to control data skipping, enabled by
default)
- `hoodie.metadata.enable` (to enable metadata table use on the read path,
enabled by default)
- `hoodie.metadata.index.column.stats.enable` (to enable column stats index
use on the read path)
+
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most:
an equality predicate on a
+high-cardinality column whose min-max range covers almost every file.
Parquet's own bloom filters cover that
+case. They are written into the Parquet file itself, and a reader consults
them to skip row groups that
+cannot contain the value being searched for.
+
+Hudi passes these through to the Parquet writer, per column, from the
**Hadoop** configuration:
+
+| key | meaning |
+| --- | --- |
+| `parquet.bloom.filter.enabled#<column>` | write a bloom filter for
`<column>` |
+| `parquet.bloom.filter.expected.ndv#<column>` | expected number of distinct
values, which sizes the filter |
+
+`<column>` is any data column you filter on by equality — not the record key,
and unrelated to the record-key
+bloom index discussed in the note below. Set the keys on the Hadoop
configuration your writer uses; from Spark
+the `spark.hadoop.` prefix forwards them:
+
+```
+--conf spark.hadoop.parquet.bloom.filter.enabled#session_id=true
+--conf spark.hadoop.parquet.bloom.filter.expected.ndv#session_id=100000
+```
+
+Give `expected.ndv` a realistic estimate for the column. Too low and the
filter saturates and stops
+eliminating anything; too high and you pay in file size for nothing.
+
+This is a write-time decision: only files written after you set it carry the
filters, so an existing table
+picks them up as it is rewritten by ongoing writes, compaction or clustering.
+
+On the read side nothing extra needs configuring for Spark 3.x. Reading the
table back through the Hudi
+datasource consults the filters, provided the query carries an equality
predicate that can be pushed down to
+the Parquet reader — a query filtering on a value no row group contains skips
those row groups entirely.
+Parquet's own read-side switch, `parquet.filter.bloom.enabled`, is left at its
default and Hudi never
+overrides it, so there is no reader-side flag to turn on.
+
+:::note
+Do not confuse the keys above with Hudi's own
`hoodie.parquet.bloom.filter.enabled`, which is a different
+feature despite the near-identical name. That config controls whether Hudi
writes a bloom filter **of record
+keys** into the file footer for use by the [bloom index](indexes.md) during
upserts; it defaults to `true`,
+applies only when meta fields are populated, and is implied anyway when
`hoodie.index.type` names a `BLOOM`
+index. It has nothing to do with per-column skipping on the read path, and
setting it does not enable the
+Parquet column filters described here.
+:::
+
+:::caution
+Hudi applies these settings reflectively, so if the Parquet version on your
classpath predates the
+`withBloomFilterEnabled` / `withBloomFilterNDV` builder methods, the keys are
**silently ignored** rather
+than rejected. If you see no change in file size or query behaviour, check
your Parquet version first.
+:::
diff --git a/website/versioned_docs/version-1.0.1/performance.md
b/website/versioned_docs/version-1.0.1/performance.md
index 7c294e57886a..384d31c16fd8 100644
--- a/website/versioned_docs/version-1.0.1/performance.md
+++ b/website/versioned_docs/version-1.0.1/performance.md
@@ -132,6 +132,56 @@ To enable Data Skipping in your queries make sure to set
following properties to
- `hoodie.metadata.enable` (to enable metadata table use on the read path,
enabled by default)
- `hoodie.metadata.index.column.stats.enable` (to enable column stats index
use on the read path)
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most:
an equality predicate on a
+high-cardinality column whose min-max range covers almost every file.
Parquet's own bloom filters cover that
+case. They are written into the Parquet file itself, and a reader consults
them to skip row groups that
+cannot contain the value being searched for.
+
+Hudi passes these through to the Parquet writer, per column, from the
**Hadoop** configuration:
+
+| key | meaning |
+| --- | --- |
+| `parquet.bloom.filter.enabled#<column>` | write a bloom filter for
`<column>` |
+| `parquet.bloom.filter.expected.ndv#<column>` | expected number of distinct
values, which sizes the filter |
+
+`<column>` is any data column you filter on by equality — not the record key,
and unrelated to the record-key
+bloom index discussed in the note below. Set the keys on the Hadoop
configuration your writer uses; from Spark
+the `spark.hadoop.` prefix forwards them:
+
+```
+--conf spark.hadoop.parquet.bloom.filter.enabled#session_id=true
+--conf spark.hadoop.parquet.bloom.filter.expected.ndv#session_id=100000
+```
+
+Give `expected.ndv` a realistic estimate for the column. Too low and the
filter saturates and stops
+eliminating anything; too high and you pay in file size for nothing.
+
+This is a write-time decision: only files written after you set it carry the
filters, so an existing table
+picks them up as it is rewritten by ongoing writes, compaction or clustering.
+
+On the read side nothing extra needs configuring for Spark 3.x. Reading the
table back through the Hudi
+datasource consults the filters, provided the query carries an equality
predicate that can be pushed down to
+the Parquet reader — a query filtering on a value no row group contains skips
those row groups entirely.
+Parquet's own read-side switch, `parquet.filter.bloom.enabled`, is left at its
default and Hudi never
+overrides it, so there is no reader-side flag to turn on.
+
+:::note
+Do not confuse the keys above with Hudi's own
`hoodie.parquet.bloom.filter.enabled`, which is a different
+feature despite the near-identical name. That config controls whether Hudi
writes a bloom filter **of record
+keys** into the file footer for use by the [bloom index](indexes.md) during
upserts; it defaults to `true`,
+applies only when meta fields are populated, and is implied anyway when
`hoodie.index.type` names a `BLOOM`
+index. It has nothing to do with per-column skipping on the read path, and
setting it does not enable the
+Parquet column filters described here.
+:::
+
+:::caution
+Hudi applies these settings reflectively, so if the Parquet version on your
classpath predates the
+`withBloomFilterEnabled` / `withBloomFilterNDV` builder methods, the keys are
**silently ignored** rather
+than rejected. If you see no change in file size or query behaviour, check
your Parquet version first.
+:::
+
## Related Resources
<h3>Blogs</h3>
diff --git a/website/versioned_docs/version-1.0.2/performance.md
b/website/versioned_docs/version-1.0.2/performance.md
index 7c294e57886a..384d31c16fd8 100644
--- a/website/versioned_docs/version-1.0.2/performance.md
+++ b/website/versioned_docs/version-1.0.2/performance.md
@@ -132,6 +132,56 @@ To enable Data Skipping in your queries make sure to set
following properties to
- `hoodie.metadata.enable` (to enable metadata table use on the read path,
enabled by default)
- `hoodie.metadata.index.column.stats.enable` (to enable column stats index
use on the read path)
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most:
an equality predicate on a
+high-cardinality column whose min-max range covers almost every file.
Parquet's own bloom filters cover that
+case. They are written into the Parquet file itself, and a reader consults
them to skip row groups that
+cannot contain the value being searched for.
+
+Hudi passes these through to the Parquet writer, per column, from the
**Hadoop** configuration:
+
+| key | meaning |
+| --- | --- |
+| `parquet.bloom.filter.enabled#<column>` | write a bloom filter for
`<column>` |
+| `parquet.bloom.filter.expected.ndv#<column>` | expected number of distinct
values, which sizes the filter |
+
+`<column>` is any data column you filter on by equality — not the record key,
and unrelated to the record-key
+bloom index discussed in the note below. Set the keys on the Hadoop
configuration your writer uses; from Spark
+the `spark.hadoop.` prefix forwards them:
+
+```
+--conf spark.hadoop.parquet.bloom.filter.enabled#session_id=true
+--conf spark.hadoop.parquet.bloom.filter.expected.ndv#session_id=100000
+```
+
+Give `expected.ndv` a realistic estimate for the column. Too low and the
filter saturates and stops
+eliminating anything; too high and you pay in file size for nothing.
+
+This is a write-time decision: only files written after you set it carry the
filters, so an existing table
+picks them up as it is rewritten by ongoing writes, compaction or clustering.
+
+On the read side nothing extra needs configuring for Spark 3.x. Reading the
table back through the Hudi
+datasource consults the filters, provided the query carries an equality
predicate that can be pushed down to
+the Parquet reader — a query filtering on a value no row group contains skips
those row groups entirely.
+Parquet's own read-side switch, `parquet.filter.bloom.enabled`, is left at its
default and Hudi never
+overrides it, so there is no reader-side flag to turn on.
+
+:::note
+Do not confuse the keys above with Hudi's own
`hoodie.parquet.bloom.filter.enabled`, which is a different
+feature despite the near-identical name. That config controls whether Hudi
writes a bloom filter **of record
+keys** into the file footer for use by the [bloom index](indexes.md) during
upserts; it defaults to `true`,
+applies only when meta fields are populated, and is implied anyway when
`hoodie.index.type` names a `BLOOM`
+index. It has nothing to do with per-column skipping on the read path, and
setting it does not enable the
+Parquet column filters described here.
+:::
+
+:::caution
+Hudi applies these settings reflectively, so if the Parquet version on your
classpath predates the
+`withBloomFilterEnabled` / `withBloomFilterNDV` builder methods, the keys are
**silently ignored** rather
+than rejected. If you see no change in file size or query behaviour, check
your Parquet version first.
+:::
+
## Related Resources
<h3>Blogs</h3>
diff --git a/website/versioned_docs/version-1.1.1/performance.md
b/website/versioned_docs/version-1.1.1/performance.md
index 1e587e563681..63228a0309f8 100644
--- a/website/versioned_docs/version-1.1.1/performance.md
+++ b/website/versioned_docs/version-1.1.1/performance.md
@@ -132,6 +132,56 @@ To enable Data Skipping in your queries make sure to set
following properties to
- `hoodie.metadata.enable` (to enable metadata table use on the read path,
enabled by default)
- `hoodie.metadata.index.column.stats.enable` (to enable column stats index
use on the read path)
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most:
an equality predicate on a
+high-cardinality column whose min-max range covers almost every file.
Parquet's own bloom filters cover that
+case. They are written into the Parquet file itself, and a reader consults
them to skip row groups that
+cannot contain the value being searched for.
+
+Hudi passes these through to the Parquet writer, per column, from the
**Hadoop** configuration:
+
+| key | meaning |
+| --- | --- |
+| `parquet.bloom.filter.enabled#<column>` | write a bloom filter for
`<column>` |
+| `parquet.bloom.filter.expected.ndv#<column>` | expected number of distinct
values, which sizes the filter |
+
+`<column>` is any data column you filter on by equality — not the record key,
and unrelated to the record-key
+bloom index discussed in the note below. Set the keys on the Hadoop
configuration your writer uses; from Spark
+the `spark.hadoop.` prefix forwards them:
+
+```
+--conf spark.hadoop.parquet.bloom.filter.enabled#session_id=true
+--conf spark.hadoop.parquet.bloom.filter.expected.ndv#session_id=100000
+```
+
+Give `expected.ndv` a realistic estimate for the column. Too low and the
filter saturates and stops
+eliminating anything; too high and you pay in file size for nothing.
+
+This is a write-time decision: only files written after you set it carry the
filters, so an existing table
+picks them up as it is rewritten by ongoing writes, compaction or clustering.
+
+On the read side nothing extra needs configuring for Spark 3.x. Reading the
table back through the Hudi
+datasource consults the filters, provided the query carries an equality
predicate that can be pushed down to
+the Parquet reader — a query filtering on a value no row group contains skips
those row groups entirely.
+Parquet's own read-side switch, `parquet.filter.bloom.enabled`, is left at its
default and Hudi never
+overrides it, so there is no reader-side flag to turn on.
+
+:::note
+Do not confuse the keys above with Hudi's own
`hoodie.parquet.bloom.filter.enabled`, which is a different
+feature despite the near-identical name. That config controls whether Hudi
writes a bloom filter **of record
+keys** into the file footer for use by the [bloom index](indexes.md) during
upserts; it defaults to `true`,
+applies only when meta fields are populated, and is implied anyway when
`hoodie.index.type` names a `BLOOM`
+index. It has nothing to do with per-column skipping on the read path, and
setting it does not enable the
+Parquet column filters described here.
+:::
+
+:::caution
+Hudi applies these settings reflectively, so if the Parquet version on your
classpath predates the
+`withBloomFilterEnabled` / `withBloomFilterNDV` builder methods, the keys are
**silently ignored** rather
+than rejected. If you see no change in file size or query behaviour, check
your Parquet version first.
+:::
+
## Related Resources
<h3>Blogs</h3>
diff --git a/website/versioned_docs/version-1.2.0/performance.md
b/website/versioned_docs/version-1.2.0/performance.md
index 7c6d489c3732..e540e770fb75 100644
--- a/website/versioned_docs/version-1.2.0/performance.md
+++ b/website/versioned_docs/version-1.2.0/performance.md
@@ -132,6 +132,56 @@ To enable Data Skipping in your queries make sure to set
following properties to
- `hoodie.metadata.enable` (to enable metadata table use on the read path,
enabled by default)
- `hoodie.metadata.index.column.stats.enable` (to enable column stats index
use on the read path)
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most:
an equality predicate on a
+high-cardinality column whose min-max range covers almost every file.
Parquet's own bloom filters cover that
+case. They are written into the Parquet file itself, and a reader consults
them to skip row groups that
+cannot contain the value being searched for.
+
+Hudi passes these through to the Parquet writer, per column, from the
**Hadoop** configuration:
+
+| key | meaning |
+| --- | --- |
+| `parquet.bloom.filter.enabled#<column>` | write a bloom filter for
`<column>` |
+| `parquet.bloom.filter.expected.ndv#<column>` | expected number of distinct
values, which sizes the filter |
+
+`<column>` is any data column you filter on by equality — not the record key,
and unrelated to the record-key
+bloom index discussed in the note below. Set the keys on the Hadoop
configuration your writer uses; from Spark
+the `spark.hadoop.` prefix forwards them:
+
+```
+--conf spark.hadoop.parquet.bloom.filter.enabled#session_id=true
+--conf spark.hadoop.parquet.bloom.filter.expected.ndv#session_id=100000
+```
+
+Give `expected.ndv` a realistic estimate for the column. Too low and the
filter saturates and stops
+eliminating anything; too high and you pay in file size for nothing.
+
+This is a write-time decision: only files written after you set it carry the
filters, so an existing table
+picks them up as it is rewritten by ongoing writes, compaction or clustering.
+
+On the read side nothing extra needs configuring for Spark 3.x. Reading the
table back through the Hudi
+datasource consults the filters, provided the query carries an equality
predicate that can be pushed down to
+the Parquet reader — a query filtering on a value no row group contains skips
those row groups entirely.
+Parquet's own read-side switch, `parquet.filter.bloom.enabled`, is left at its
default and Hudi never
+overrides it, so there is no reader-side flag to turn on.
+
+:::note
+Do not confuse the keys above with Hudi's own
`hoodie.parquet.bloom.filter.enabled`, which is a different
+feature despite the near-identical name. That config controls whether Hudi
writes a bloom filter **of record
+keys** into the file footer for use by the [bloom index](indexes.md) during
upserts; it defaults to `true`,
+applies only when meta fields are populated, and is implied anyway when
`hoodie.index.type` names a `BLOOM`
+index. It has nothing to do with per-column skipping on the read path, and
setting it does not enable the
+Parquet column filters described here.
+:::
+
+:::caution
+Hudi applies these settings reflectively, so if the Parquet version on your
classpath predates the
+`withBloomFilterEnabled` / `withBloomFilterNDV` builder methods, the keys are
**silently ignored** rather
+than rejected. If you see no change in file size or query behaviour, check
your Parquet version first.
+:::
+
## Related Resources
<h3>Blogs</h3>
