yihua opened a new issue, #908: URL: https://github.com/apache/hudi-rs/issues/908
### Feature Description The per-record cost removed for record index writes in #872 is not specific to the record index. In `apache-avro` 0.21, `to_avro_datum` validates the value (building a `ResolvedSchema`) and then encodes it (building another), and `from_avro_datum` builds one more on decode. Each build walks the full `HoodieMetadataRecord` schema, including every index payload type and the column stats wrapper unions, and the crate exposes no entry point that takes an already resolved schema. Every metadata table record encoded or decoded through the generic path pays that walk. Paths that still pay it once per record: - **Record index decode**: `decode_record_index_entry`, called per entry in `index/record.rs` `apply_hfile_records` when loading record locations. Tagging N keys on an upsert decodes N entries, which mirrors the write side fixed in #872. - **Column stats decode**: `decode_column_stats_entry`, called per existing stats record in `write/metadata.rs` `apply_column_stats_record` when merging stats on write. `records.rs` `decode_avro_value` does the same for files partition records. - **Stats encode**: `encode_column_stats` / `encode_partition_stats`, called once per (file, column) and (partition, column). The multiplier is lower than one per row, but a wide table writing many files reaches the same order of records. `encode_files_record` runs once per partition and is unlikely to matter. Proposed: 1. Decode record index entries straight from Avro binary, held to the generic decoder in tests like the encoder. The decoder honors the container's writer schema for tables with mixed writers, so the direct path applies only when that schema matches the vendored layout, with the generic decode as the fallback. 2. Measure the column stats paths before changing them. Their record shape is not fixed (min/max values are a union over the wrapper types, and V1 and V2 differ), so a hand-written encoder there costs more than the record index one did. The alternative is to resolve the schema once and reuse it across records, either through an `apache-avro` API that accepts resolved schemata or a minimal encoder/decoder of our own. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
