alitrack commented on PR #819: URL: https://github.com/apache/hudi-rs/pull/819#issuecomment-6242076097
Status note from the consumer side (duckdb-hudi, building index decoders on this API): three findings from validating `read_metadata_table_partition_arrow` against the repo's own fixtures that may help the review. 1. **Per-partition reads do not hit the `avro_to_arrow` `AvroSchema::Ref` gap.** The v2 reader decodes HFile→Arrow internally, so `column_stats` (272 rows) and `partition_stats` (48 rows) on `v9_txns_simple_meta` come back as clean batches even though the MDT union-of-records schema uses named-type references. This PR is self-contained — no avro-Ref fix is needed first. 2. **Point lookups with `keys` work end-to-end on a sharded partition.** The record-index partition stores the **bare record key** (e.g. `TXN-011`, not SHA-1 hashed), so routing (`slices_for_keys`) and the HFile key match line up: a lookup for `TXN-011` returns exactly one row (opening 1 of the 10 shards); a lookup for a non-existent key returns an empty batch, not an error. This is what the "matched as stored" doc comment promises, now measured. 3. **Caveat for API callers (pre-existing kernel behaviour, not introduced by this PR):** file groups that are log-only with zero non-tombstone log records fail with `No schema available for merge output` (`reader_v2/engine.rs:629`). That is what a full read of the `record_index` partition hits on the V8 trips table, and of the `secondary_index_*` partitions on the v9 fixture. Worth a `# Error` note on the public method, or special-casing the all-tombstone log case in the merge-schema lookup. The sample tables in this repo reproduce all of the above without any external data. Would be great to get a reviewer on this — it unblocks the DuckDB extension's record-index / bloom / point-lookup surface. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
