cshuo opened a new issue, #20668:
URL: https://github.com/apache/hudi/issues/20668

   ### Description
   
   Spark snapshot queries on a Merge-on-Read table can return incomplete 
results when metadata bloom filter data skipping is enabled.
   
   Metadata bloom filters cover base files only. However, 
`BloomFiltersIndexSupport` returns candidate base filenames, and 
`HoodieFileIndex` uses those candidates to decide whether to retain an entire 
file slice. This can incorrectly exclude:
   
   - A base+log slice when the queried record key exists only in its logs and 
the base bloom returns negative.
   - A log-only slice, which has no base filename to contribute to the 
candidate set.
   
   The records remain readable when data skipping is disabled.
   
   ### Steps to reproduce
   
   Create a MOR table with:
   
   ```properties
   hoodie.datasource.write.recordkey.field=_row_key
   hoodie.datasource.write.partitionpath.field=partition
   hoodie.metadata.enable=true
   hoodie.metadata.index.bloom.filter.enable=true
   hoodie.metadata.index.bloom.filter.column.list=_row_key
   hoodie.compact.inline=false
   hoodie.compact.schedule.inline=false
   ```
   
   Disable other query indexes to isolate metadata bloom pruning, including 
bucket query pruning, record-level indexes, and column/partition statistics.
   
   Perform these writes without compaction:
   
   1. Use a single-bucket writer and `insert_overwrite` to write key `a` into 
partition `base-and-log`, and key `c` into partition `base-only`.
   2. Upsert a new key `b` into partition `base-and-log`. The existing bucket 
routes it into the existing file group's log.
   3. Use the `INMEMORY` writer index and `insert` to write key `d` into a new 
partition `log-only`, creating a slice without a base file.
   
   Verify the resulting layout:
   
   | Partition | Base records | Log records |
   |---|---|---|
   | `base-and-log` | `a` | `b` |
   | `base-only` | `c` | — |
   | `log-only` | — | `d` |
   
   Both base-file blooms must be present and return negative for `b` and `d`.
   
   Run snapshot queries with data skipping enabled and disabled:
   
   ```scala
   def query(keys: Seq[String], skipping: Boolean): DataFrame = {
     spark.read.format("hudi")
       .option("hoodie.datasource.query.type", "snapshot")
       .option("hoodie.enable.data.skipping", skipping.toString)
       .option("hoodie.bucket.query.index", "false")
       .load(basePath)
       .filter(col("_row_key").isin(keys: _*))
       .select("_row_key")
   }
   ```
   
   ### Expected and actual results
   
   | Predicate | Expected / skipping disabled | Skipping enabled |
   |---|---|---|
   | `_row_key = 'b'` | `b` | Empty |
   | `_row_key = 'd'` | `d` | Empty |
   | `_row_key IN ('b', 'd')` | `b, d` | Empty |
   | `_row_key = 'c'` | `c` | `c` |
   
   Read-optimized queries correctly exclude `b` and `d`, since those queries 
only read base files.
   
   ### Root cause
   
   `BloomFiltersIndexer` skips delta write stats and non-base files, so the 
metadata bloom filters do not describe keys newly inserted into logs.
   
   `BloomFiltersIndexSupport.computeCandidateFileNames` nevertheless derives 
candidates only from base files. `HoodieFileIndex.filterFileSlices` then drops 
any slice whose base/log filenames do not intersect those candidates. A 
negative base bloom therefore becomes an incorrect negative result for the 
whole slice.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to