rangareddy commented on code in PR #19590:
URL: https://github.com/apache/hudi/pull/19590#discussion_r3764333086
##########
website/docs/performance.md:
##########
@@ -132,6 +132,48 @@ To enable Data Skipping in your queries make sure to set
following properties to
- `hoodie.metadata.enable` (to enable metadata table use on the read path,
enabled by default)
- `hoodie.metadata.index.column.stats.enable` (to enable column stats index
use on the read path)
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most:
an equality predicate on a
+high-cardinality column whose min-max range covers almost every file.
Parquet's own bloom filters cover that
Review Comment:
Good catch, and it needed answering rather than hand-waving — but the
underlying premise turns out to be testable, and the answer is better than
"reframe as write-side only".
Hudi has an end-to-end test for exactly this, `TestHoodieParquetBloomFilter`
in `hudi-spark`. It sets the two write keys on `jsc.hadoopConfiguration`, then
deliberately disables the other two filter levels so nothing else can cause a
skip:
```scala
sparkSession.sql("set parquet.filter.columnindex.enabled=false")
sparkSession.sql("set parquet.filter.stats.enabled=false")
```
writes through `format("hudi")`, and reads back through
`sparkSession.read.format("hudi")`:
```scala
.filter("bloom_col = '3'") // value absent -> asserts 0 row groups
scanned (Spark >= 3)
.filter("bloom_col = '2'") // value present -> asserts 1
```
parameterised over BULK_INSERT, INSERT, UPSERT and INSERT_OVERWRITE on COW.
So the Hudi Spark read path does consult Parquet bloom filters out of the box
on Spark 3.x, and the row-group skip is asserted, not assumed. Note also what
the test implies about your `parquet.filter.bloom.enabled` question: it
switches off *stats* and *columnindex* but never touches the bloom switch, so
bloom filtering is on at its parquet-mr default — and `grep` confirms Hudi
never sets that key anywhere.
You were still right that the section said "a reader consults them" and then
documented only write-side keys, leaving a reader to guess. Added in `1d0…` →
now `41a3a0d`:
> On the read side nothing extra needs configuring for Spark 3.x. Reading
the table back through the Hudi datasource consults the filters, provided the
query carries an equality predicate that can be pushed down to the Parquet
reader — a query filtering on a value no row group contains skips those row
groups entirely. Parquet's own read-side switch,
`parquet.filter.bloom.enabled`, is left at its default and Hudi never overrides
it, so there is no reader-side flag to turn on.
One deliberate omission: I do **not** name `TestHoodieParquetBloomFilter` in
the docs page. That is the same objection you raised on #19486 — a user-facing
reference to a test class rots silently on a rename — so the evidence lives in
the commit message and this thread instead. I had it in the page for one
revision and took it out for that reason.
I have left the Spark-4 / vectorized-reader question alone rather than
guessing: the test gates its assertion on `spark.version >= 3`, so Spark 3.x is
what is actually pinned, and that is all the text claims.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
