HippoBaro opened a new pull request, #12367:
URL: https://github.com/apache/arrow-rs/pull/12367

   # Which issue does this PR close?
   
   - Closes #12364
   
   # Rationale for this change
   
   This provides the low-level support requested in 
[#12364](https://github.com/apache/arrow-rs/issues/12364) for attaching 
application metadata to existing Parquet structures using [Binary Protocol 
Extensions](https://github.com/apache/parquet-format/blob/master/BinaryProtocolExtensions.md).
 The crate supplies coordinates and payload transport; applications interpret 
the bytes.
   
   # What changes are included in this PR?
   
   This PR introduces three connected pieces: a shared coordinate system, a 
writer mechanism for attaching extension payloads, and a reader mechanism for 
observing them. Both mechanisms use the same coordinates to describe where a 
payload is written or encountered, giving applications a consistent view of its 
context in either direction.
   
   The foundation is **shared coordinate and context machinery** used by both 
serialization and decoding. `MetadataSite` identifies the encoded root, 
original row-group/column/page positions where available, and a relative Thrift 
path resolved on request. These are structural coordinates, not file offsets. 
Named field-ID constants let applications interpret paths without duplicating 
the codecs' numeric definitions. Writers and streaming readers reuse the same 
bounded ancestry implementation.
   
   On the **write side**, applications register a `MetadataExtensionHook` with 
targets declaring the roots, positional contexts, and root-only or descendant 
coverage of interest. At each matching ordinary struct, `on_site` receives its 
coordinates and can return bytes to append as field `32767` before STOP, or 
`None` to omit the extension. The hook can inspect the detailed path for finer 
selection. Borrowed payloads are consumed before the next callback. Union 
envelopes are excluded; their ordinary struct members remain eligible.
   
   On the **read side**, applications opt in through the existing reader 
options. Matching extension fields encountered during normal decoding invoke 
`MetadataExtensionListener::on_extension` with their bytes and the same 
coordinate model. Observation adds no I/O and does not change projection or 
pruning. Callback errors propagate to the enclosing operation.
   
   ## Cost model
   
   Tracking context and ancestry has a non-negligable cost/ The implementation 
attempts to be zero-cost for non-participating users: unobserved codecs avoid 
extension ancestry tracking, callback locking, and payload allocation; 
root-only selections also avoid ancestry tracking. Buffered readers borrow 
payloads, streaming readers buffer observed payloads as they arrive, and 
detailed reader paths are discovered on-demand.
   
   The aim is to keep added work local to selected contexts and extension data. 
Some bookkeeping remains: optional bindings occupy space, and active scopes 
need state to supply coordinates. A writer hook can also be invoked and return 
no payload. Costs therefore depend on the selected scopes visited as well as 
payloads processed. 
   
   ## Example
   
   `parquet/examples/metadata_extensions.rs` attaches a UUID-prefixed 
measurement unit to a `speed` field's `SchemaElement`. A named observer 
validates the declared unit during footer loading, accepting `m/s` or rejecting 
an incompatible expectation before reading rows. Payload identification and 
interpretation remain application code.
   
   ```sh
   cargo run -p parquet --example metadata_extensions --features experimental
   ```
   
   # Are these changes tested?
   
   All tests passing.New tests focus on the codecs and shared 
coordinate/selection machinery, including wire framing, borrowing, bounds, 
error recovery, and encryption. 
   
   # Are there any significant user-facing changes?
   
   The extension APIs and field-ID namespace are behind `experimental`. 
`ChunkReader::try_get_buffered_range` is deliberately ungated: its default 
returns `None`, while memory-backed implementations can expose resident bytes 
for slice-based page-header decoding without I/O or copying. Other readers 
retain the streaming fallback.
   
   The crate does not impose a payload format or automatically preserve 
extensions during rewriting. Each configuration accepts one hook or listener 
with its complete target list. No standalone page-header decoder or 
footer-independent reader is added.
   
   A standardized extension mechanism does not guarantee that every reader 
handles every legal placement. The [base of this 
branch](https://github.com/apache/arrow-rs/blob/9f45c44cd0ed19f7bb5c93eda6b7e163c5f4b06d/parquet/src/parquet_thrift.rs#L478-L485)
 rejects fields inside nominally empty structs, including ordinary union-member 
structs; this PR adds unknown-field handling there. Separately, [#8714's 
compatibility 
discussion](https://github.com/apache/arrow-rs/pull/8714#discussion_r2467093357)
 reports older readers attempting UTF-8 decoding while skipping binary fields. 
Files containing extensions may therefore be rejected by existing readers even 
when the placement and payload are valid. Compatibility should be verified for 
the reader implementations, versions, and extension locations an application 
uses.
   
   # AI usage disclosure
   
   AI tools assisted with implementation and refactoring, tests, the executable 
example, documentation, and code review. I fully own the changes, including 
design, scope decisions and implementation details. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to