CTTY opened a new pull request, #3734: URL: https://github.com/apache/iceberg-rust/pull/3734
## Which issue does this PR close? - Closes #TODO (issue to be filed before this leaves draft). Related avro-rs issue: TODO. **Stacked on #3733.** The first commit here is that PR; review the second commit (`perf(manifest): reuse parsed Avro writer schemas across manifests`). ## What changes are included in this PR? `Manifest::try_from_avro_bytes` built an `apache_avro::Reader` per manifest, and `Reader` parses the writer schema embedded in the file header every time: JSON decode, `Schema::parse` and name resolution. The manifests of one table version carry byte-identical `avro.schema`, and profiling `plan_files` on a 601-manifest table attributes about half of the per-manifest cost to that parse. apache-avro 0.22 has no way to hand `Reader` an already-parsed writer schema. New `avro::container::ObjectContainerFile` reads the object container format itself (magic, header metadata, codec, sync marker, blocks) and decodes records with `GenericDatumReader`, which is the same `SchemaAwareDeserializer` that `Reader::into_deser_iter` uses, with the same `human_readable` default. The parsed writer schema comes from a `ParsedCache` (from #3733) keyed by the header's `avro.schema` bytes, so a repeated header costs a hash of the schema text. The distinct-key count per snapshot is small: partition specs × format versions × distinct writers, since adding a column does not change a manifest's Avro schema. Kept from `Reader`: per-block sync marker check; a record that consumes no bytes of a non-empty block is an error; a block's record count never sizes an allocation beyond the block's length. The fallback for manifests whose writer schema defines a named type more than once (older iceberg-rust releases) now rewrites the schema JSON before parsing, and the cached entry records the repeated names so each such manifest is still logged. Behavior differences from `Reader`, as currently written: malformed files fail with this crate's wording (kind stays `DataInvalid`); a non-`bytes` user-metadata value is an error instead of a warning; a partial varint at end of file is an error instead of EOF. The right long-term home for this is avro-rs accepting a pre-parsed writer schema; the module doc says so and this module goes away when it does. Measured on top of #3670, planning a 601-manifest pyiceberg-written table with `plan_files`, 4 cpu workers, fresh `Table` per iteration, median of 20: TPC-H SF10 7.0 → 2.8 ms, SF100 65.4 → 25.1 ms. ### Open items while in draft - [ ] `ObjectContainerFile::records()` returns a `Vec`; make it an iterator. As written, a large manifest is fully decoded into `Vec<ManifestEntryVn>` before conversion, roughly doubling peak memory versus `Reader::into_deser_iter`, and #3670 runs several manifests at once. - [ ] Decide on the two stricter-than-`Reader` behaviors above; I lean towards matching `Reader` so no manifest that read before fails after. - [ ] File the avro-rs issue for a pre-parsed writer schema entry point and link it from the module doc. - [ ] Add snappy and zstandard to the round-trip test (both features are enabled; current test covers null and deflate). - [ ] Re-measure after the iterator change. ## Are these changes tested? Unit tests in `avro/container.rs`: round trip through `apache_avro::Writer` with null and deflate codecs, one and two blocks, with user metadata, and an empty file; header errors (bad magic, unsupported codec, truncated sync marker); block errors (block past end of file, mismatched sync marker, negative count, count beyond the block, record that consumes no bytes); cache behavior (same bytes hit, different bytes miss, repeated named types rewritten once and cached, original error reported when repeated definitions differ). In `spec/manifest/mod.rs`: two manifests with byte-identical writer schemas decode identically with the second served from the cache; a manifest whose writer schema differs by one partition field is not served the cached schema; the existing repeated-named-type fixture and truncated-header test. `cargo test -p iceberg --lib`: 2297 passed. ## AI Disclosure Written with an AI assistant (Claude via Kiro); I reviewed and tested every change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
