slachiewicz opened a new issue, #901: URL: https://github.com/apache/incubator-xtable/issues/901
`ParquetConversionSource` builds the table schema from exactly one file's footer. `createInternalTableFromFile` (ParquetConversionSource.java:88) reads a single `LocatedFileStatus`, and every caller feeds it one file selected by `getMostRecentParquetFile` — `getCurrentSnapshot` via `getMostRecentTable` (:206), and the table-change path at :190-191. Two consequences for a Parquet directory treated as a table: 1. **Schema evolution loses columns.** If a column was added (or dropped) partway through the table's history, the reported schema is whichever generation the newest-by-modification-time file happens to belong to. Files from other generations are then listed as data files whose columns do not all exist in the read schema (or which lack columns the schema promises). Modification time is also not a stable proxy for "newest schema": a backup/restore, `distcp`, or object-store copy rewrites mtimes, so the same directory can report different schemas before and after a copy. 2. **A Hive-partitioned directory can report a partition field with no backing column.** `partitionSpecExtractor.spec(schema)` resolves the configured partition paths against that single footer's schema. When the partition column exists only in the directory structure (`region=eu/...`) and not inside the files — common for Hive-written data — the partition field resolves against nothing, while the values remain derivable from the paths. A possible direction for (1): merge the footers of all listed files into the read schema (fields present in any file, with a hard error when two files disagree on a field's type), rather than trusting one file. For (2): synthesize the partition column into the read schema, typed from the observed path values, when the configured partition path is absent from the footers. Happy to provide a small fixture directory (two file generations plus a Hive-partitioned layout) if useful. *This issue was created with AI assistance.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
