brkyvz opened a new pull request, #585: URL: https://github.com/apache/parquet-format/pull/585
<!-- Thanks for opening a pull request! If you're new to Parquet-Format, information on how to contribute can be found here: https://parquet.apache.org/docs/contribution-guidelines/contributing Please open a GitHub issue for this pull request: https://github.com/apache/parquet-format/issues/new/choose and format pull request title as below: GH-${GITHUB_ISSUE_ID}: ${SUMMARY} or simply use the title below if it is a minor issue: MINOR: ${SUMMARY} --> ### Rationale for this change Introduces a new type called File as a typed FileReference. The design document is [here](https://docs.google.com/document/d/1AiwrstqkwkBoOZqgOkm9JGwSMcNeHyLR7EEj1CVqpZQ/edit?tab=t.0#heading=h.k8qyue4jj4rn). The motivation is as follows: ``` Unstructured data ingestion is getting extremely popular with the advances in Generative AI. Today, our only means of dealing with unstructured data is to store it as a binary blob inside Parquet, or point to files that exist in some object store with a string. These solutions fail to address these use cases, because of scalability, usability, and governance issues. We would like to introduce a new logical type annotation in Parquet called “File” for storing a struct that contains a path reference to a file with additional metadata. This reference may be to a file that exists (or expected to exist) in storage at a given path. We’d like to define the minimum required list of fields that would allow a client to correctly read the referenced data. Any additional metadata can be optionally stored by engines and table formats as necessary adjacent to this type. ``` ### What changes are included in this PR? Introduces the specification for FileType. ### Do these changes have PoC implementations? Yes: https://github.com/apache/parquet-java/pull/3608/changes <!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the actual Github issue id. --> <!-- Closes #${GITHUB_ISSUE_ID} --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
