cshuo opened a new issue, #20535: URL: https://github.com/apache/hudi/issues/20535
When a column is added through schema evolution, older files may not contain it in their Parquet footers. If the new column is indexed, those files have no column stats for it. Partition stats for that column are then invalidated because file coverage is incomplete. Queries remain correct, but partition pruning for the new column may be reduced until the older files are replaced or complete stats become available. **Proposed improvement:** Explore whether files written before the column was added can safely be represented with all-null column stats. This would allow partition stats to remain available when combined with stats from newer files. The implementation should distinguish a column absent from the file schema from other cases of missing statistics, and account for schema defaults, nested fields, and MOR/log files. **Suggested validation:** - Add a column to a table with existing files, then write new files in the same partition. - Verify the resulting column and partition stats, including min/max and null/value counts. - Compare query results and pruning behavior with data skipping enabled and disabled for COW and MOR tables. Related PR: https://github.com/apache/hudi/pull/20328 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
