NoahKusaba opened a new issue, #75: URL: https://github.com/apache/datafusion-iceberg/issues/75
### Is your feature request related to a problem or challenge? apache/iceberg-rust#2904 changes `TableScanBuilder::build` so that a scan that doesn't pin a snapshot projects the table's current schema, not the schema of the snapshot it reads. Before it, a scan of a table whose schema changed after its last write resolved columns against the older schema. That is the root cause of #42. That PR checks the snapshot and schema a scan resolves with unit tests. End-to-end checks of the rows read back were asked to live here as sqllogictests rather than as integration tests in iceberg-rust ([review comment](https://github.com/apache/iceberg-rust/pull/2904#discussion_r4256575583)). ### Describe the solution you'd like Once this crate depends on an iceberg-rust release that includes apache/iceberg-rust#2904, add sqllogictests that change a table's schema after its last write, with no write since, and check what queries return: - **Column added:** `SELECT *` returns the new column as NULL for every existing row. `WHERE c IS NULL` matches every row, and `WHERE c = <value>` matches none. This covers #42. - **Column dropped:** selecting the dropped column fails with a column-not-found error, and `SELECT *` no longer returns it. - **Column dropped and added back under the same name:** the new column reads as NULL, not the old column's values. Data files are matched by field ID, and the new field ID isn't in any file. - **Column renamed:** the values read back under the new name. The sqllogictest engine here runs only DataFusion, and DataFusion SQL can't change an Iceberg table's schema. These tests need a way to do it, for example a setup step that calls `Transaction::update_schema()` or a Spark engine, as the README lists. ### Additional context - #42: `SELECT *` fails after a column is added until the next write to the table. - #57 is related but not covered here: a provider built before the schema change keeps its cached schema. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
