imonteroq commented on issue #20296: URL: https://github.com/apache/hudi/issues/20296#issuecomment-6345619696
We hit this failure mode on `hudi-spark4.1-bundle:1.2.0` without any `column.list` change or schema evolution on the queried columns. A conjunctive filter `identifier = 1 AND date = 100` intermittently returned 0 rows on a COW table where each single-column predicate matched and the data was present: ``` identifier = 1 → 1 date = 100 → 3 identifier = 1 AND date = 100 → 0 (correct: 1) ``` ~40% of test-suite runs before instrumentation; 0 reproductions in ~2,700 instrumented queries after. The `column_stats` partition is read via `getRecordsByKeyPrefixes`, and we saw recurring `Stream not closed`/`IOException` on `.col-stats-*.log` files in the same runs, so a plausible trigger is a transiently missing `(column, file)` stats record (partial MDT read or a data-commit/stats-commit visibility gap), though we could not isolate it. We then reproduced the mechanism deterministically: write file1 with `hoodie.metadata.index.column.stats.column.list=identifier`, expand to `identifier,date`, write file2. File1 shows stats for `identifier` but no `date` record, and the conjunction drops it: `catalog=0, load=0, skipping-off=1` (COW, `hoodie.write.table.version=6`, non-partitioned, S3). The point for this issue: the missing-stats state isn't only reachable via config changes; it can arise transiently in the wild, and when it does the result is silent row loss on conjunctive queries. This argues for the trigger-agnostic guard in #20328 (retain files whose conjunct stat columns show missing stats) rather than a fix scoped to index expansion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
