voonhous opened a new issue, #19487: URL: https://github.com/apache/hudi/issues/19487
### Describe the problem you faced On a table using the **partition-level** simple bucket index, any query whose data filters do not constrain the bucket hash field returns **zero rows**. `PartitionBucketIndexSupport.computeCandidateFileNames` wraps its per-partition result in `Option.apply(...)` unconditionally, so "I cannot prune anything" is reported as `Some(Set.empty)` instead of `Option.empty`: https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/PartitionBucketIndexSupport.scala#L69-L79 The parent `BucketIndexSupport.computeCandidateFileNames` gets this right and returns `Option.empty` in the same situation (`BucketIndexSupport.scala:89-90`). ### Failure chain 1. A predicate that does not reference the hash field (e.g. `where name = 'x'` when the hash field is `id`) falls through `getBucketsBySingleHashFields` to `case _`, which does `setUntil(numBuckets)` -- every bucket matches (`BucketIndexSupport.scala:200-203`). 2. `filterQueriesWithBucketHashField` sees `cardinality == bucketNumber` and returns `None`, logging *"the query predicates do not include equality expressions for all the hashing fields, fall back to the other indices"* (`BucketIndexSupport.scala:122-124`). 3. In `PartitionBucketIndexSupport` that `None` hits the `else` branch, yielding `Seq()` for every partition, and the whole thing is wrapped as `Option.apply(Set())` = `Some(Set())`. 4. `HoodieFileIndex.lookupCandidateFilesInMetadataTable` tests `prunedFileNames.nonEmpty`, which is `Option.isDefined` -- so it returns `Some(Set())` and **short-circuits** the secondary, expression, bloom-filter and column-stats indices that were supposed to run next. 5. `HoodieFileIndex.filterFileSlices` (lines 300-303) sees `candidateFilesNamesOpt.isDefined` and `contains` false for every file, so it filters out **all** file slices. The `dataFilters.isEmpty` guard at `HoodieFileIndex.scala:266` only protects unfiltered scans, so this is reachable for ordinary queries. ### Why it was not caught `TestBucketIndexSupport` pins the correct behaviour with 14 `fallback = true` assertions, but `TestPartitionBucketIndexSupport` has none. The end-to-end suite `TestInsertTableWithPartitionBucketIndex` ("Test BucketID Pruning With Partition Bucket Index") only ever queries `where id = '1111'` -- the hash field itself -- so the fallback path is never exercised. Found while reviewing #19471, which re-enables the previously skipped `TestPartitionBucketIndexSupport` assertions on Spark 4. ### Expected behavior `PartitionBucketIndexSupport.computeCandidateFileNames` should return `Option.empty` when no partition produced a bucket bitmap, so `HoodieFileIndex` falls through to the remaining indices instead of pruning the table away. ### Reproduction Add this to `TestPartitionBucketIndexSupport` (`B` is not a hash field, so the index must fall back) -- it fails today: ```scala val bitmap = bucketIndexSupport.filterQueriesWithBucketHashField(Seq(resolvedBExpr)) assert(bitmap.isEmpty) // passes: index correctly declines to prune val candidate = bucketIndexSupport.computeCandidateFileNames( fileIndex, Seq(resolvedBExpr), Seq(), input, false) assert(candidate.isEmpty) // FAILS: returns Some(Set()) instead of None ``` Observed against Spark 4.0.2: ``` FILTER[B = 'abc'] PartitionBucketIndexSupport -> Some(size=0) Set() BucketIndexSupport -> None (fall back, all files scanned) FILTER[A = 5 Or B = 'abc'] -> same divergence FILTER[A = C] -> same divergence ``` ### Environment Description * Hudi version: master (regression introduced by #13060, [HUDI-8990]) * Spark version: reproduced on 4.0.2, not version specific * Running on Docker? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
