zn-danielbariudin opened a new pull request, #4384:
URL: https://github.com/apache/iceberg-python/pull/4384

   Closes #3662
   
   # Rationale for this change
   
   Every manifest in a table carries the same Avro schema string in its header, 
but `AvroFileHeader.get_schema()` parsed and converted that string to an 
Iceberg `Schema` on every file open. During scan planning this repeats the same 
conversion once per manifest.
   
   This caches the conversion by schema string with `functools.lru_cache`, so 
each distinct schema is converted once per process.
   
   - `Schema` is a frozen model, so sharing the cached instance between readers 
is safe.
   - `lru_cache` keeps at most 128 schema strings; a process sees only a few 
distinct manifest schemas.
   
   Builds on #3663 by @vishnuprakaz (picked up with his 👍 on the issue); he is 
credited as co-author on the commit.
   
   ### Benchmark
   
   `plan_files()` on an unpartitioned table with 150 manifests (local 
`SqlCatalog`, 7 runs in one process), Python 3.14.7, Apple M5 Pro:
   
   | | Before | After |
   |---|---|---|
   | Schema conversions in one `plan_files()` | 151 | 1 |
   | `plan_files()` median | ~56 ms | ~30 ms |
   | `plan_files()` first call | ~96 ms | ~67 ms |
   
   <details>
   <summary>Benchmark script</summary>
   
   ```python
   import statistics
   import sys
   import tempfile
   import time
   
   import pyarrow as pa
   from pyiceberg.avro.file import AvroSchemaConversion
   from pyiceberg.catalog.sql import SqlCatalog
   
   manifests = int(sys.argv[1])
   repeats = 7
   
   warehouse = tempfile.mkdtemp()
   catalog = SqlCatalog("bench", uri=f"sqlite:///{warehouse}/catalog.db", 
warehouse=f"file://{warehouse}")
   catalog.create_namespace("ns")
   table = catalog.create_table("ns.t", schema=pa.schema([("id", pa.int64())]))
   for i in range(manifests):
       table.append(pa.table({"id": pa.array([i], pa.int64())}))
   
   calls = 0
   original = AvroSchemaConversion.avro_to_iceberg
   
   
   def counting(self, avro_schema):
       global calls
       calls += 1
       return original(self, avro_schema)
   
   
   AvroSchemaConversion.avro_to_iceberg = counting
   
   timings = []
   for run in range(repeats):
       start = time.perf_counter()
       tasks = list(table.scan().plan_files())
       timings.append((time.perf_counter() - start) * 1000)
       if run == 0:
           first_run_calls = calls
   
   print(f"manifests={manifests} data_files={len(tasks)}")
   print(f"conversions: first plan_files={first_run_calls}, all {repeats} 
runs={calls}")
   print(f"plan_files ms: first={timings[0]:.1f} 
median={statistics.median(timings):.1f} min={min(timings):.1f}")
   ```
   
   Run with `uv run python bench_plan_files.py 150`.
   </details>
   
   ## Are these changes tested?
   
   Yes. Added `test_get_schema_reuses_conversion_for_identical_schema_strings` 
in `tests/avro/test_file.py`: it checks the converted schema, that two headers 
with the same schema string return the same instance, and that a different 
string returns a different one. The full unit suite passes locally.
   
   ## Are there any user-facing changes?
   
   No API changes. Scan planning is faster on tables with many manifests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to