zn-danielbariudin opened a new pull request, #4384:
URL: https://github.com/apache/iceberg-python/pull/4384
Closes #3662
# Rationale for this change
Every manifest in a table carries the same Avro schema string in its header,
but `AvroFileHeader.get_schema()` parsed and converted that string to an
Iceberg `Schema` on every file open. During scan planning this repeats the same
conversion once per manifest.
This caches the conversion by schema string with `functools.lru_cache`, so
each distinct schema is converted once per process.
- `Schema` is a frozen model, so sharing the cached instance between readers
is safe.
- `lru_cache` keeps at most 128 schema strings; a process sees only a few
distinct manifest schemas.
Builds on #3663 by @vishnuprakaz (picked up with his 👍 on the issue); he is
credited as co-author on the commit.
### Benchmark
`plan_files()` on an unpartitioned table with 150 manifests (local
`SqlCatalog`, 7 runs in one process), Python 3.14.7, Apple M5 Pro:
| | Before | After |
|---|---|---|
| Schema conversions in one `plan_files()` | 151 | 1 |
| `plan_files()` median | ~56 ms | ~30 ms |
| `plan_files()` first call | ~96 ms | ~67 ms |
<details>
<summary>Benchmark script</summary>
```python
import statistics
import sys
import tempfile
import time
import pyarrow as pa
from pyiceberg.avro.file import AvroSchemaConversion
from pyiceberg.catalog.sql import SqlCatalog
manifests = int(sys.argv[1])
repeats = 7
warehouse = tempfile.mkdtemp()
catalog = SqlCatalog("bench", uri=f"sqlite:///{warehouse}/catalog.db",
warehouse=f"file://{warehouse}")
catalog.create_namespace("ns")
table = catalog.create_table("ns.t", schema=pa.schema([("id", pa.int64())]))
for i in range(manifests):
table.append(pa.table({"id": pa.array([i], pa.int64())}))
calls = 0
original = AvroSchemaConversion.avro_to_iceberg
def counting(self, avro_schema):
global calls
calls += 1
return original(self, avro_schema)
AvroSchemaConversion.avro_to_iceberg = counting
timings = []
for run in range(repeats):
start = time.perf_counter()
tasks = list(table.scan().plan_files())
timings.append((time.perf_counter() - start) * 1000)
if run == 0:
first_run_calls = calls
print(f"manifests={manifests} data_files={len(tasks)}")
print(f"conversions: first plan_files={first_run_calls}, all {repeats}
runs={calls}")
print(f"plan_files ms: first={timings[0]:.1f}
median={statistics.median(timings):.1f} min={min(timings):.1f}")
```
Run with `uv run python bench_plan_files.py 150`.
</details>
## Are these changes tested?
Yes. Added `test_get_schema_reuses_conversion_for_identical_schema_strings`
in `tests/avro/test_file.py`: it checks the converted schema, that two headers
with the same schema string return the same instance, and that a different
string returns a different one. The full unit suite passes locally.
## Are there any user-facing changes?
No API changes. Scan planning is faster on tables with many manifests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]