rangareddy commented on issue #15970:
URL: https://github.com/apache/hudi/issues/15970#issuecomment-4889971615
This issue was reviewed as part of the JIRA-migrated backlog triage and
validated on Hudi 1.2.0 (Spark 3.4).
Findings: the column-stats index is correct for MERGE_ON_READ tables across
all four cases in this checklist (tested with
`hoodie.metadata.index.column.stats.enable=true` on column `fare`):
1) one col-stats value per file slice
2) merged base+log slices: base file {10,20,30} plus a log update setting
fare=1000 (a value present only in the log). The metadata-table column_stats
records one entry per data file (base min=10/max=30; log min=1000/max=1000), a
data-skipping query `fare >= 900` correctly returns the log-only record (the
file group is not wrongly pruned), and `fare > 5000` prunes to 0.
3) delete log blocks: deleting the high-value record d (fare=1000) via a
delete operation, then `fare >= 900` returns 0 (total=3) - the delete log block
is handled correctly, no stale read and the col_stats read does not error.
4) custom delete payloads: soft-deleting d via `_hoodie_is_deleted` (honored
by DefaultHoodieRecordPayload) gives the same correct result (total=3,
`fare>=900`=0).
In every case col stats never cause a false prune / data loss - the base
file's max is retained and deletes are applied at merge time. Validation passes
on current Hudi; no code change needed.
Closing as validated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
