andygrove opened a new issue, #6822:
URL: https://github.com/apache/datafusion-comet/issues/6822
### Describe the bug
A `VERSION AS OF` read of a snapshot taken before a column was renamed
returns wrong results from the native Iceberg scan. The renamed column comes
back as NULL. When two columns swap names, each one returns the other's values.
Spark returns the snapshot's data, and reads of the current snapshot are
correct.
`CometIcebergNativeScan.serializePartitions` gives a task with no deletes
the current table schema, and a task with deletes `FileScanTask.schema()`,
which is also the current schema. It switches to the scan schema only when the
scan reads a column the current schema no longer has (`hasHistoricalColumns`).
A renamed column keeps its field id, so that check doesn't fire. iceberg-rust
then names each output column after the current schema, and Comet matches those
columns to the snapshot names Spark expects by name. A renamed column finds no
match and reads as NULL. After a swap, `projectFieldIds` also resolves each
output name in the task schema first, where the name now belongs to the other
column.
### Steps to reproduce
```sql
CREATE TABLE cat.db.t (id INT, a STRING, b STRING) USING iceberg;
INSERT INTO cat.db.t SELECT CAST(id AS INT), concat('A', id), concat('B',
id) FROM range(20);
-- <s> is the snapshot id after the insert
ALTER TABLE cat.db.t RENAME COLUMN a TO z;
SELECT id, a FROM cat.db.t VERSION AS OF <s> ORDER BY id;
```
On `main` at 00a4b422f0 and on `branch-1.1` at e9efd9f764 (Spark 4.1,
Iceberg 1.11):
| After the snapshot | Query | Spark | Comet |
| --- | --- | --- | --- |
| `a` renamed to `z` | `SELECT id, a ... VERSION AS OF <s>` | `[0,A0] [1,A1]
[2,A2] …` | `[0,null] [1,null] [2,null] …` |
| `a` and `b` swapped through a temporary name | `SELECT id, a, b ...
VERSION AS OF <s>` | `[0,A0,B0] [1,A1,B1] …` | `[0,B0,A0] [1,B1,A1] …` |
| `a` and `b` swapped | `SELECT id, b ... VERSION AS OF <s>` | `[0,B0]
[1,B1] …` | `[0,A0] [1,A1] …` |
The results are the same when the snapshot has position deletes. Spark 3.4
with Iceberg 1.5.2 gives the same results too. I checked that through #6725
with its pruning turned off, which takes the same path as `main` for these
queries.
### Expected behavior
The native scan returns the snapshot's data, as Spark does. A `VERSION AS
OF` task needs a schema with the snapshot's column names whenever they differ
from the current schema's, not only when a column was dropped.
### Additional context
Found while reviewing #6725. That PR reads with Spark's pruned scan schema
by default, which carries the snapshot's names, so these queries match Spark
with it. The bug stays on `branch-1.1`, with
`spark.comet.scan.icebergNative.nestedSchemaPruning.enabled=false`, and for
tasks that keep the full schema because the query pruned away a nested
partition source or equality-delete key. A swap where the scan also has to
append a partition source on one of the swapped columns fails at planning with
#6725 (`Invalid schema: multiple fields for name b`), which is discussed on
that PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
