vishnuprakaz opened a new pull request, #3663:
URL: https://github.com/apache/iceberg-python/pull/3663
Closes #3662
# Rationale for this change
`AvroFileHeader.get_schema()` re-parses the embedded Avro schema and
converts it to an Iceberg schema on every file
open. All manifests in a table share the same schema, so during scan
planning we redo that identical conversion
once per manifest for no reason.
Caching it (keyed on the schema string) means each distinct schema is
converted only once.
I timed `scan().plan_files()` on unpartitioned tables of a few sizes,
`main` vs this branch (median of 10 warm runs):
| manifests | before | after | speedup |
|---:|---:|---:|---:|
| 10 | 5.6 ms | 3.1 ms | 1.8x |
| 50 | 29.4 ms | 15.6 ms | 1.9x |
| 150 | 86.2 ms | 48.0 ms | 1.8x |
| 400 | 233.9 ms | 128.8 ms | 1.8x |
Roughly 1.8x throughout, and since the wasted work is per-manifest, larger
tables save more. Results are identical.
This is a narrow, unpartitioned table; a partitioned one has a wider
manifest schema and would gain at least as
much.
Uses `functools.lru_cache`, like the existing `_cached_resolve_s3_region`
in `io/pyarrow.py`, so no new dependency.
`Schema` is immutable and already shared across scan threads, so sharing
one cached instance is safe.
## Are these changes tested?
Yes. A unit test in `tests/avro/test_file.py` checks the cached schema is
correct, that the same schema string
returns the same object, and that a different string returns a different
schema. There's also a benchmark in
`tests/benchmark/test_avro_schema_cache_benchmark.py` (run with `-m
benchmark`) for the numbers above. Existing
avro/table/manifest tests pass.
## Are there any user-facing changes?
No, same behavior, just faster scan planning on tables with many manifests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
