paulcaron16k commented on issue #3758:
URL:
https://github.com/apache/iceberg-python/issues/3758#issuecomment-5211634083
Retested against `7d0f5031` (`version = "0.12.0"`). Still reproduces, and I
was able to narrow it usefully.
## It is specific to temporal transforms
Same script, only the partition transform changed.
`append([("a",1),("b",1)])` then `upsert([("a",2)])`:
| partition spec | result |
|---|---|
| `identity(ts)` | ✅ `[('a', 2), ('b', 1)]` |
| `truncate(k, 1)` | ✅ `[('a', 2), ('b', 1)]` |
| `bucket(k, 4)` | ✅ `[('a', 2), ('b', 1)]` |
| `year(ts)` | ❌ `[('a', 1), ('a', 2), ('b', 1), ('b', 1)]` |
| `month(ts)` | ❌ same |
| `day(ts)` | ❌ same |
| `hour(ts)` | ❌ same |
So it is not "partitioned" in general, and not "non-identity" either —
`truncate` and `bucket` are both fine. Only the four temporal transforms fail,
which are the ones whose partition value is an integer ordinal derived from a
timestamp source column.
That fits the predicate I mentioned above. Traced during a run partitioned
by `day(ts)`:
```
Or(EqualTo(term=Reference(name='ts'), literal=LongLiteral(20455)), ...)
```
The *source column* (`ts`, a timestamp) compared against a *partition
ordinal* (`20455`, days since epoch). For `identity` the two coincide, and for
`truncate`/`bucket` the projection is evidently still usable; for the temporal
transforms the resulting predicate cannot select the manifest, so
`_existing_manifests` keeps it whole.
## Minimum condition
One manifest containing **both** a row being replaced and a row that is not.
- append 1 row, upsert it → ✅ correct. Every entry in the manifest is
deleted, so the manifest is dropped entirely and the stale path is never taken.
- append 2 rows, upsert 1 → ❌ one entry survives, so the manifest has to be
rewritten, the evaluator does not match, and it is appended verbatim with
**both** entries — beside the newly written ones.
## Minimal reproduction
```python
import datetime as dt
import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import DayTransform
from pyiceberg.types import IntegerType, NestedField, StringType,
TimestampType
warehouse = tempfile.mkdtemp()
catalog = SqlCatalog("c", uri=f"sqlite:///{warehouse}/c.db",
warehouse=f"file://{warehouse}")
catalog.create_namespace("db")
table = catalog.create_table(
"db.t",
schema=Schema(
NestedField(1, "k", StringType(), required=False),
NestedField(2, "v", IntegerType(), required=False),
NestedField(3, "ts", TimestampType(), required=False),
),
partition_spec=PartitionSpec(PartitionField(3, 1000, DayTransform(),
"p")),
properties={"format-version": "2"},
)
schema = pa.schema(
[pa.field("k", pa.string()), pa.field("v", pa.int32()), pa.field("ts",
pa.timestamp("us"))]
)
when = dt.datetime(2026, 1, 6, 12)
def batch(pairs):
return pa.table(
{
"k": [k for k, _ in pairs],
"v": pa.array([v for _, v in pairs], type=pa.int32()),
"ts": [when] * len(pairs),
},
schema=schema,
)
table.append(batch([("a", 1), ("b", 1)])) # one manifest, two rows
table.refresh()
table.upsert(batch([("a", 2)]), join_cols=["k"]) # replace one of them
table.refresh()
result = table.scan().to_arrow()
print(sorted(zip(result["k"].to_pylist(), result["v"].to_pylist())))
# 0.11.1 -> [('a', 2), ('b', 1)]
# 0.12.0 -> [('a', 1), ('a', 2), ('b', 1), ('b', 1)]
```
Happy to test a patch against these cases if that helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
