vishnuprakaz opened a new issue, #3662:
URL: https://github.com/apache/iceberg-python/issues/3662
### Feature Request / Improvement
### Feature Request / Improvement
`AvroFileHeader.get_schema()` (`pyiceberg/avro/file.py`) runs the full
`avro_to_iceberg` conversion every time an
Avro file is opened.
**The inefficiency**
- Every manifest under a spec embeds an *identical* Avro schema string.
- So during scan planning, that same conversion is repeated **once per
manifest**.
- The cost grows with the manifest count, even though only a couple of
distinct schemas are ever involved.
**Proposed fix**
- The conversion depends only on the schema string, so the result can be
cached (keyed on that string).
**Measured impact**
- `scan().plan_files()` on an unpartitioned 150-manifest table: **~86 ms →
~48 ms (~1.8× faster)**.
- The saving grows as the number of manifests increases.
I am willing to contribute for this improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
