CTTY opened a new pull request, #3734:
URL: https://github.com/apache/iceberg-rust/pull/3734

   ## Which issue does this PR close?
   
   - Closes #TODO (issue to be filed before this leaves draft). Related avro-rs 
issue: TODO.
   
   **Stacked on #3733.** The first commit here is that PR; review the second 
commit (`perf(manifest): reuse parsed Avro writer schemas across manifests`).
   
   ## What changes are included in this PR?
   
   `Manifest::try_from_avro_bytes` built an `apache_avro::Reader` per manifest, 
and `Reader` parses the writer schema embedded in the file header every time: 
JSON decode, `Schema::parse` and name resolution. The manifests of one table 
version carry byte-identical `avro.schema`, and profiling `plan_files` on a 
601-manifest table attributes about half of the per-manifest cost to that 
parse. apache-avro 0.22 has no way to hand `Reader` an already-parsed writer 
schema.
   
   New `avro::container::ObjectContainerFile` reads the object container format 
itself (magic, header metadata, codec, sync marker, blocks) and decodes records 
with `GenericDatumReader`, which is the same `SchemaAwareDeserializer` that 
`Reader::into_deser_iter` uses, with the same `human_readable` default. The 
parsed writer schema comes from a `ParsedCache` (from #3733) keyed by the 
header's `avro.schema` bytes, so a repeated header costs a hash of the schema 
text. The distinct-key count per snapshot is small: partition specs × format 
versions × distinct writers, since adding a column does not change a manifest's 
Avro schema.
   
   Kept from `Reader`: per-block sync marker check; a record that consumes no 
bytes of a non-empty block is an error; a block's record count never sizes an 
allocation beyond the block's length. The fallback for manifests whose writer 
schema defines a named type more than once (older iceberg-rust releases) now 
rewrites the schema JSON before parsing, and the cached entry records the 
repeated names so each such manifest is still logged.
   
   Behavior differences from `Reader`, as currently written: malformed files 
fail with this crate's wording (kind stays `DataInvalid`); a non-`bytes` 
user-metadata value is an error instead of a warning; a partial varint at end 
of file is an error instead of EOF.
   
   The right long-term home for this is avro-rs accepting a pre-parsed writer 
schema; the module doc says so and this module goes away when it does.
   
   Measured on top of #3670, planning a 601-manifest pyiceberg-written table 
with `plan_files`, 4 cpu workers, fresh `Table` per iteration, median of 20: 
TPC-H SF10 7.0 → 2.8 ms, SF100 65.4 → 25.1 ms.
   
   ### Open items while in draft
   
   - [ ] `ObjectContainerFile::records()` returns a `Vec`; make it an iterator. 
As written, a large manifest is fully decoded into `Vec<ManifestEntryVn>` 
before conversion, roughly doubling peak memory versus 
`Reader::into_deser_iter`, and #3670 runs several manifests at once.
   - [ ] Decide on the two stricter-than-`Reader` behaviors above; I lean 
towards matching `Reader` so no manifest that read before fails after.
   - [ ] File the avro-rs issue for a pre-parsed writer schema entry point and 
link it from the module doc.
   - [ ] Add snappy and zstandard to the round-trip test (both features are 
enabled; current test covers null and deflate).
   - [ ] Re-measure after the iterator change.
   
   ## Are these changes tested?
   
   Unit tests in `avro/container.rs`: round trip through `apache_avro::Writer` 
with null and deflate codecs, one and two blocks, with user metadata, and an 
empty file; header errors (bad magic, unsupported codec, truncated sync 
marker); block errors (block past end of file, mismatched sync marker, negative 
count, count beyond the block, record that consumes no bytes); cache behavior 
(same bytes hit, different bytes miss, repeated named types rewritten once and 
cached, original error reported when repeated definitions differ).
   
   In `spec/manifest/mod.rs`: two manifests with byte-identical writer schemas 
decode identically with the second served from the cache; a manifest whose 
writer schema differs by one partition field is not served the cached schema; 
the existing repeated-named-type fixture and truncated-header test.
   
   `cargo test -p iceberg --lib`: 2297 passed.
   
   ## AI Disclosure
   
   Written with an AI assistant (Claude via Kiro); I reviewed and tested every 
change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to