HippoBaro opened a new pull request, #12367: URL: https://github.com/apache/arrow-rs/pull/12367
# Which issue does this PR close? - Closes #12364 # Rationale for this change This provides the low-level support requested in [#12364](https://github.com/apache/arrow-rs/issues/12364) for attaching application metadata to existing Parquet structures using [Binary Protocol Extensions](https://github.com/apache/parquet-format/blob/master/BinaryProtocolExtensions.md). The crate supplies coordinates and payload transport; applications interpret the bytes. # What changes are included in this PR? This PR introduces three connected pieces: a shared coordinate system, a writer mechanism for attaching extension payloads, and a reader mechanism for observing them. Both mechanisms use the same coordinates to describe where a payload is written or encountered, giving applications a consistent view of its context in either direction. The foundation is **shared coordinate and context machinery** used by both serialization and decoding. `MetadataSite` identifies the encoded root, original row-group/column/page positions where available, and a relative Thrift path resolved on request. These are structural coordinates, not file offsets. Named field-ID constants let applications interpret paths without duplicating the codecs' numeric definitions. Writers and streaming readers reuse the same bounded ancestry implementation. On the **write side**, applications register a `MetadataExtensionHook` with targets declaring the roots, positional contexts, and root-only or descendant coverage of interest. At each matching ordinary struct, `on_site` receives its coordinates and can return bytes to append as field `32767` before STOP, or `None` to omit the extension. The hook can inspect the detailed path for finer selection. Borrowed payloads are consumed before the next callback. Union envelopes are excluded; their ordinary struct members remain eligible. On the **read side**, applications opt in through the existing reader options. Matching extension fields encountered during normal decoding invoke `MetadataExtensionListener::on_extension` with their bytes and the same coordinate model. Observation adds no I/O and does not change projection or pruning. Callback errors propagate to the enclosing operation. ## Cost model Tracking context and ancestry has a non-negligable cost/ The implementation attempts to be zero-cost for non-participating users: unobserved codecs avoid extension ancestry tracking, callback locking, and payload allocation; root-only selections also avoid ancestry tracking. Buffered readers borrow payloads, streaming readers buffer observed payloads as they arrive, and detailed reader paths are discovered on-demand. The aim is to keep added work local to selected contexts and extension data. Some bookkeeping remains: optional bindings occupy space, and active scopes need state to supply coordinates. A writer hook can also be invoked and return no payload. Costs therefore depend on the selected scopes visited as well as payloads processed. ## Example `parquet/examples/metadata_extensions.rs` attaches a UUID-prefixed measurement unit to a `speed` field's `SchemaElement`. A named observer validates the declared unit during footer loading, accepting `m/s` or rejecting an incompatible expectation before reading rows. Payload identification and interpretation remain application code. ```sh cargo run -p parquet --example metadata_extensions --features experimental ``` # Are these changes tested? All tests passing.New tests focus on the codecs and shared coordinate/selection machinery, including wire framing, borrowing, bounds, error recovery, and encryption. # Are there any significant user-facing changes? The extension APIs and field-ID namespace are behind `experimental`. `ChunkReader::try_get_buffered_range` is deliberately ungated: its default returns `None`, while memory-backed implementations can expose resident bytes for slice-based page-header decoding without I/O or copying. Other readers retain the streaming fallback. The crate does not impose a payload format or automatically preserve extensions during rewriting. Each configuration accepts one hook or listener with its complete target list. No standalone page-header decoder or footer-independent reader is added. A standardized extension mechanism does not guarantee that every reader handles every legal placement. The [base of this branch](https://github.com/apache/arrow-rs/blob/9f45c44cd0ed19f7bb5c93eda6b7e163c5f4b06d/parquet/src/parquet_thrift.rs#L478-L485) rejects fields inside nominally empty structs, including ordinary union-member structs; this PR adds unknown-field handling there. Separately, [#8714's compatibility discussion](https://github.com/apache/arrow-rs/pull/8714#discussion_r2467093357) reports older readers attempting UTF-8 decoding while skipping binary fields. Files containing extensions may therefore be rejected by existing readers even when the placement and payload are valid. Compatibility should be verified for the reader implementations, versions, and extension locations an application uses. # AI usage disclosure AI tools assisted with implementation and refactoring, tests, the executable example, documentation, and code review. I fully own the changes, including design, scope decisions and implementation details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
