sivabalan narayanan created HUDI-5245: -----------------------------------------
Summary: Honor pruned partitions while looking up in col stats partition in MDT Key: HUDI-5245 URL: https://issues.apache.org/jira/browse/HUDI-5245 Project: Apache Hudi Issue Type: Improvement Components: metadata Reporter: sivabalan narayanan When looking up in col stats for data skipping, we are passing in only the list of columns in the predicate. We don't leverage the pruned list of partitions in this call. For eg, if there are 1000 partitions and 100 cols, and only 10 partitions are matched after pruning, exiting call will fetch 100 cols * 1000 partitions = 10k entries from col_stats partition in MDT to do file skipping. where as if we wire in pruned list of partitions, then we only need to do file skipping from 1000 entries. -- This message was sent by Atlassian Jira (v8.20.10#820010)