anoopj opened a new issue, #2844: URL: https://github.com/apache/iceberg-rust/issues/2844
### Apache Iceberg Rust version 0.6.0 (latest version) ### Describe the bug plan_files() returns an error for any scan with a predicate, even one that touches no partition column if the snapshot contains a manifest written under a historical partition spec whose source column is no longer in the current schema. The unfiltered scan of the same table succeeds, so the data is readable; adding any filter makes the table's historical data unqueryable. This is a legal table state in format v2 and later (including v3): a column may be dropped once it is no longer used by the default spec, but historical specs referencing it are retained in table metadata because existing manifests still use them. Related: #2842 (the same table state panics the manifests metadata table; fix in #2843). This issue is the scan-planning counterpart and is independent of that fix. ### To Reproduce 1. Create a v2 table (v3 behaves identically) with schema `(id int, is_test boolean)` and spec `identity(is_test)` (spec 0), and write a snapshot so a manifest with spec 0 exists. 2. Evolve the spec to remove the `is_test partition` field (spec 1 becomes default). 3. Drop the `is_test column` from the schema. 4. Scan with any predicate on a live column, e.g. id > 0, and call plan_files(). Step 4 returns: Unexpected => No column with source column id 2 in schema [...] An unfiltered scan of the same table succeeds and returns the expected `FileScanTask`s (with the correct partition values, since manifest reading resolves types from the manifest's own embedded metadata). ### Expected behavior The scan should succeed. Partition pruning is an optimization: a manifest whose spec can't be fully resolved should at worst not be pruned, never cause planning to fail. Two levels of fix: 1. Stopgap: when a spec's partition type can't be resolved against the current schema, skip partition pruning for that spec's manifests (treat the partition filter as always-true). This should be strictly safe since a predicate cannot reference a dropped column, so the skipped filter could never have pruned based on the missing field anyway; remaining row-level filtering is unaffected. I'd implement this as a resolvability pre-check (all of the spec's source IDs present in the schema) rather than by matching the error, so it composes with the longer-term fix. 2. Longer term: make partition_type succeed for this state: derive the output type from the transform where it doesn't depend on the source (bucket, year/month/day/hour, void) and degrade to the v3 unknown type only for identity/truncate. This is the approach iceberg-java is taking in apache/iceberg#17262 (context: apache/iceberg#16958); for rust it also depends on adding the v3 unknown type, which I'll file separately. With that in place, pruning keeps working on the still-resolvable fields and the stopgap becomes unnecessary. I can contribute the stopgap fix. ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
