Hello, The FILE PR [1] is in good shape — path/size/offset/etag are in, and there's an active thread on adding content_type. Following last week's sync and the discussing in the comments of the PR, we took a step back and looked at the difference in complexity between having two separate types: one narrowly focused on referencing a file and later introducing one that can accommodate inline/in-file/packed representations and it all came down to the addition of a single field (along with how fields are interpreted). Rather than have multiple types that significantly overlap in capability and representation, we think it make sense to provide a single type that can address all necessary features. The changes in short:
- make path optional — an absent path means a self-reference to the current file (already raised in review; makes self-referencing files trivially moveable) - add inline for storing bytes directly in the value - generalize etag into a self-describing checksum (it carries an etag as one case) FILE then spans inline/in-file/external as one type. Defining the full set now avoids a second type later that every reader and writer would have to handle. Details in the "FILE — complete type" tab of the design doc [2]. Feedback welcome. Regards, Alkis [1] https://github.com/apache/parquet-format/pull/585 [2] https://docs.google.com/document/d/1AiwrstqkwkBoOZqgOkm9JGwSMcNeHyLR7EEj1CVqpZQ/edit
