paulcaron16k opened a new issue, #3758:
URL: https://github.com/apache/iceberg-python/issues/3758
### Apache Iceberg version
main (reproduced at `154288fb`; the affected files are byte-identical at
`7d0f5031`, which is `version = "0.12.0"`)
### Please describe the bug 🐞
On a **partitioned** table, `upsert` leaves the row it replaced *and*
duplicates a row it never touched. No error is raised.
`0.11.1` is correct. An unpartitioned table is correct on both, so the
partition spec is required to reproduce.
## Reproduction
```python
import tempfile, pathlib, datetime as dt
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.schema import Schema
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.transforms import DayTransform
from pyiceberg.types import NestedField, StringType, IntegerType,
TimestampType
wh = pathlib.Path(tempfile.mkdtemp())
cat = SqlCatalog("c", uri=f"sqlite:///{wh}/c.db", warehouse=f"file://{wh}")
cat.create_namespace("db")
schema = Schema(
NestedField(1, "k", StringType(), required=False),
NestedField(2, "v", IntegerType(), required=False),
NestedField(3, "ts", TimestampType(), required=False),
)
spec = PartitionSpec(
PartitionField(source_id=3, field_id=1000, transform=DayTransform(),
name="ts_day")
)
t = cat.create_table("db.t", schema=schema, partition_spec=spec,
properties={"format-version": "2"})
arrow = pa.schema([pa.field("k", pa.string()), pa.field("v", pa.int32()),
pa.field("ts", pa.timestamp("us"))])
base = dt.datetime(2026, 1, 6, 12)
def rows(pairs):
return pa.table({"k": [p[0] for p in pairs],
"v": pa.array([p[1] for p in pairs], type=pa.int32()),
"ts": [base] * len(pairs)}, schema=arrow)
t.append(rows([("a", 1), ("b", 1)]))
t.refresh()
t.upsert(rows([("a", 2)]), join_cols=["k"])
t.refresh()
arr = t.scan().to_arrow()
print(sorted(zip(arr["k"].to_pylist(), arr["v"].to_pylist())))
print("data files:", len(t.inspect.files().to_pylist()))
```
| version | output | data files |
|---|---|---|
| 0.11.1 | `[('a', 2), ('b', 1)]` — correct | 2 |
| main | `[('a', 1), ('a', 2), ('b', 1), ('b', 1)]` | 3 |
Two problems in one result: the superseded `('a', 1)` survives beside its
replacement, and `('b', 1)` — untouched by the upsert — is duplicated.
## Suspected cause
`_OverwriteFiles._existing_manifests` keeps a manifest its evaluator does
not match, whole:
```python
manifest_evaluators: dict[int, Callable[[ManifestFile], bool]] =
KeyDefaultDict(self._build_manifest_evaluator)
...
if not manifest_evaluators[manifest_file.partition_spec_id](manifest_file):
existing_files.append(manifest_file)
continue
```
The predicate comes from `_build_delete_files_partition_predicate`, built
from the deleted files' partition values. For this upsert the join column is
`k`, which is not a partition field, so the resulting predicate cannot select
the manifest holding the rows being replaced. That manifest is appended intact,
while the rewritten file and the new row are added alongside it, so a reader
sees both copies.
I have not tried to determine whether the right fix is to widen the
predicate, to fall back to a full scan when the predicate cannot be projected
onto a spec, or to match deleted entries directly — happy to test a patch.
## Knock-on effect
The symptom most users will hit first is the *next* upsert, which fails in
`upsert_util.get_rows_to_update`:
```
ValueError: Target table has duplicate rows, aborting upsert
```
That guard is doing its job. It is detecting real duplicates — just ones the
previous `upsert` created.
## Why this may be worth looking at before 0.12
`version` on main is already `0.12.0`, and the failure is silent: a
merge-style ingest into a partitioned table accumulates duplicate rows with no
error until a later upsert happens to trip the duplicate check.
### Willingness to contribute
Would be interested in contributing a fix to Apache Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
