raufer opened a new issue, #4294:
URL: https://github.com/apache/iceberg-python/issues/4294
### Apache Iceberg version
0.12.0 (latest release). `main` at e3564a54c1 has the same code.
### Please describe the bug
`Transaction.table_metadata` recomputes on every read:
```python
@property
def table_metadata(self) -> TableMetadata:
return update_table_metadata(self._table.metadata, self._updates)
```
`update_table_metadata` ends in `new_metadata.model_copy(deep=True)`, so
every read deep-copies the whole `TableMetadata`, snapshot list included. One
commit reads the property tens of times, so its cost grows linearly with the
snapshots the table keeps, whatever rows it writes. Each commit also adds a
snapshot, so N commits to one table cost O(N²) in total.
#2674 and #3301 hoisted repeated reads out of loops. The property itself is
still uncached.
### What we measured
One live table in a development environment: 126 partitions, 1.09M rows,
snapshots expired after 24 h. Committed to about 225 times an hour, it reached
thousands of snapshots.
| Measurement | Result |
|---|---|
| Metadata at about 2,770 snapshots | 2.6 MB |
| Metadata at 5,410 snapshots | 5.1 MB |
| One deep copy at 5,410 snapshots, on a laptop | 0.6–1.5 s |
| Reads of the property in one `dynamic_partition_overwrite` of 500 rows |
51 |
| `deepcopy` calls in that commit at 192 snapshots (cProfile) | 865k |
| One commit, local SQL catalog, at 51 / 144 / 184 snapshots | 0.6 s / 2.0 s
/ 3.3 s |
| The same data in 20 commits / in one commit, local SQL catalog | 66 s /
5.5 s |
cProfile puts most of the commit in that `model_copy(deep=True)`. We did not
time a whole commit at the 2,770 or 5,410 depths.
We work around it by batching commits and expiring snapshots. Neither
removes the repeated copies.
### Reproduction
Each timed commit appends one row. `reads` counts calls to
`update_table_metadata` from `pyiceberg.table`.
```sh
uv run --isolated --no-project --python 3.12 --with
'pyiceberg[sql-sqlite,pyarrow]==0.12.0' python repro.py
```
```python
import tempfile
import time
from unittest import mock
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.table import update_table_metadata
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType
warehouse = tempfile.mkdtemp()
catalog = SqlCatalog("repro", uri=f"sqlite:///{warehouse}/catalog.db",
warehouse=f"file://{warehouse}")
catalog.create_namespace("ns")
table = catalog.create_table(
"ns.t",
schema=Schema(
NestedField(1, "key", StringType(), required=False),
NestedField(2, "value", LongType(), required=False),
),
partition_spec=PartitionSpec(PartitionField(source_id=1, field_id=1000,
transform=IdentityTransform(), name="key")),
)
def rows(key: str, n: int) -> pa.Table:
return pa.table({"key": pa.array([key] * n, pa.string()), "value":
pa.array(range(n), pa.int64())})
for target in (0, 100, 200, 400, 800):
while len(table.metadata.snapshots) < target:
table.append(rows("other", 1))
size = len(table.metadata.model_dump_json())
snapshots = len(table.metadata.snapshots)
with mock.patch("pyiceberg.table.update_table_metadata",
wraps=update_table_metadata) as reads:
start = time.perf_counter()
table.append(rows("timed", 1))
elapsed = time.perf_counter() - start
print(f"snapshots={snapshots:4d} metadata={size / 1e6:.2f} MB
append={elapsed * 1000:7.1f} ms reads={reads.call_count}", flush=True)
```
One run, one timed append per depth, Python 3.12.11, pyarrow 26.0.1, on a
laptop:
```text
snapshots= 0 metadata=0.00 MB append= 80.0 ms reads=19
snapshots= 100 metadata=0.09 MB append= 32.9 ms reads=22
snapshots= 200 metadata=0.15 MB append= 55.8 ms reads=22
snapshots= 400 metadata=0.28 MB append= 158.8 ms reads=22
snapshots= 800 metadata=0.55 MB append= 315.2 ms reads=22
```
### The fix exists
Two PRs fix this, and the stale bot closed both unmerged:
- #3302, "Cache Transaction.table_metadata between staged updates". Per
#3784, a maintainer approved it in April.
- #3784, "perf: cache Transaction.table_metadata between reads", which
carries #3302 forward. It reports 22 → 2 `update_table_metadata` calls per
append, 4.8× at 500 snapshots, and a production writer back from 10.3 s to 2.9
s.
Could a maintainer reopen #3784, or say what path would land it?
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
