rangareddy commented on issue #17354: URL: https://github.com/apache/hudi/issues/17354#issuecomment-5351237741
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8816). **Findings: needs a runtime check; I could not settle it by reading the code.** The plumbing to select which columns get indexed is well developed on `master` - `HoodieTableMetadataUtil.getColumnsToIndex` has four overloads (`:1080`, `:1113`, `:1122`, `:1131`) resolving from table config, metadata config, schema and index version, and the column-stats record generation at `:880-895` takes an explicit `columnsToIndex` list. What that does not tell me is the actual question here: when the configured column set **changes** on an existing table, what happens to the column_stats and partition_stats entries already written for the old set? Specifically - are stale columns pruned, is the partition rebuilt, or is a mixed-state index left behind that a reader could then use for skipping? That is a write / change-config / write / read sequence, not something the source answers. Whoever picks this up should start by establishing the current behaviour on a recent build, because the answer determines whether this is a feature to add or a correctness bug to fix. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
