dontirun opened a new issue, #1309:
URL: https://github.com/apache/iceberg-go/issues/1309
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
### Version
- **Working:** v0.5.0-rc (pseudo-version
`v0.5.0-rc0.0.20260302222559-f192c8e504eb`)
- **Broken:** v0.6.0
### Summary
After upgrading from v0.5 to v0.6.0, calling `AddFiles()` on a table that
has had a prior `Delete()` snapshot produces a manifest list file that Redshift
Spectrum cannot parse:
```
ERROR: Wrong type in Avro file.
Field: partitions. Expected: 12. Got: 7
location: avro_utils.hpp:55
```
The Delete snapshot itself is readable. A standalone `AddFiles` on a fresh
table (no prior Delete) is also readable. But when `AddFiles`' fast-append
snapshot producer inherits manifests from a prior Delete (overwrite) snapshot
into its manifest list, the resulting Avro encoding of the `partitions` field
becomes incompatible with Redshift's reader.
Reverting to v0.5 resolves the issue.
### Table Setup
- Glue catalog (AWS)
- Format version: 2
- Unpartitioned table (also reproduces with partitioned)
### Reproduction Steps
**Step 1**: Create table, APPEND data via `AddFiles` → queryable via
Redshift ✓
**Step 2**: `Delete()` with a filter (separate transaction) → queryable via
Redshift ✓
**Step 3**: `AddFiles()` again (separate transaction, new data) → commit
succeeds, but **Redshift returns XX000**
### Code
```go
// Transaction 1: initial append (works fine)
tbl, _ := catalog.LoadTable(ctx, identifier)
txn := tbl.NewTransaction()
txn.AddFiles(ctx, initialFiles, nil, false)
txn.Commit(ctx)
// Transaction 2: delete (works fine, queryable after)
tbl, _ = catalog.LoadTable(ctx, identifier)
txn = tbl.NewTransaction()
txn.Delete(ctx, iceberg.EqualTo(iceberg.Reference("status"), "DELETED"), nil)
txn.Commit(ctx)
// Transaction 3: append again (commit succeeds but Redshift can't read the
result)
tbl, _ = catalog.LoadTable(ctx, identifier)
txn = tbl.NewTransaction()
txn.AddFiles(ctx, newFiles, nil, false)
txn.Commit(ctx) // manifest list produced here is unreadable
```
### Test Matrix
| Scenario
| Result |
|
--------------------------------------------------------------------------- |
-------------------- |
| `AddFiles` on fresh table (no prior Delete)
| **PASS** - queryable |
| `Delete` after prior `AddFiles` data (separate transactions)
| **PASS** -queryable |
| `AddFiles` after prior `Delete` (separate transactions)
| **FAIL** - XX000 |
| `Delete` + `AddFiles` in same transaction (table has prior `AddFiles`
data) | **FAIL** - XX000 |
### Possible Root Cause
I suspect the issue may be in how the
[`fastAppend`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/transaction.go#L50)
snapshot producer (used by
[`AddFiles`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/transaction.go#L980))
writes the manifest list when inheriting manifests from a prior
[`mergeOverwrite`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/transaction.go#L59)
([`Delete`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/transaction.go#L1318))
snapshot.
[`Delete`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/transaction.go#L1318)
uses the
[`mergeOverwrite`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/transaction.go#L59)
snapshot producer, which writes manifests that Redshift reads correctly. The
subsequent
[`AddFiles`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/transaction.go#L980)
uses
[`appendSnapshotProducer`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/transaction.go#L157),
whose
[`existingManifests()`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/snapshot_producers.go#L90)
method inherits the parent snapshot's manifests. When the parent is a
Delete/overwrite snapshot, the inherited manifests are included in a new
manifest list file written by `fastAppend`.
The resulting manifest list Avro file has an encoding of the `partitions`
field that Redshift's reader rejects (`Expected: 12. Got: 7`). I believe his
only happens when the inherited manifests come from a `mergeOverwrite`
producer. Inheriting from another `fastAppend` snapshot works fine.
### Potentially Relevant Changes (v0.5 → v0.6)
- [**PR #871**](https://github.com/apache/iceberg-go/pull/871): Avro library
replacement (`hamba/avro` to `twmb/avro`). The OCF writer changed from
`ocf.NewEncoderWithSchema` with `FullSchemaMarshaler` to `ocf.NewWriter` with
`fileSchema.String()`
- [**PR #869**](https://github.com/apache/iceberg-go/pull/869): `fast-append
must inherit all parent manifests unconditionally`. Changed
[`fastAppendFiles.existingManifests()`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/snapshot_producers.go#L90)
to no longer filter manifests
- [**PR
#799**](https://github.com/apache/iceberg-go/pull/799)/[**#806**](https://github.com/apache/iceberg-go/pull/806):
`AddFiles` now uses
[`appendSnapshotProducer()`](https://github.com/apache/iceberg-go/blob/v0.6.0/table/transaction.go#L157)
instead of hardcoded `fastAppend()`
The manifest list schema has a `partitions` field typed as `union[null,
array<record>]`. When the fast-append writer serializes a manifest list that
includes entries originally produced by the overwrite producer, the Avro schema
JSON or binary encoding of this field differs from what Redshift expects (type
12 vs type 7).
### Additional Note
I tried querying via Athena instead of Redshift for one of the
Delete+AddFiles same-transaction cases. Athena does not error, but returns
**wrong data**. Columns from the appended file come back as NULL/empty even
though the underlying parquet files contain values. Rows from the Delete's
internal copy-on-write rewrite read correctly, but rows from the `AddFiles`
portion do not. This further confirms something is wrong with the manifest
metadata produced by `AddFiles` when inheriting from a Delete snapshot.
### Environment
- Go 1.22
- AWS Glue catalog
- Queried via Amazon Redshift Spectrum
- Table format version 2
- Parquet files with embedded Iceberg field IDs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
