Sagar Sumit created HUDI-7954:
---------------------------------
Summary: Fix data skipping with secondary index when there are no
log files
Key: HUDI-7954
URL: https://issues.apache.org/jira/browse/HUDI-7954
Project: Apache Hudi
Issue Type: Task
Reporter: Sagar Sumit
Assignee: Sagar Sumit
Fix For: 1.0.0-beta2, 1.0.0
When there are no log files in index, then the lookup returns no secondary keys
or candidate files, because of a bug - `logRecordsMap` is empty in this code
and base file records are ignored -
[https://github.com/apache/hudi/blob/70f44efe298771fcef9d029820a9b431e1ff165c/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java#L970]
Even though current tests for pruning asserts the filtered files count < total
data files count. It is weak in the sense that it does not filtered files count
> 0, and hence the assertion passed even when filtered files count = 0.
Ultimately, all files were getting scanned. We should fix this behavior
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
