Davis-Zhang-Onehouse opened a new pull request, #20684: URL: https://github.com/apache/hudi/pull/20684
### Describe the issue this Pull Request addresses Adds RFC-118, the design for ingesting unstructured files (PDF, images, video) announced by S3/GCS event notifications into Hudi tables with Hudi Streamer, and for keeping per-input embedding tables over them up to date. Feature issue: #20683. ### Summary and Changelog - `rfc/rfc-118/rfc-118.md`: the RFC. It proposes one new source, `UnstructuredDataSource`, with two jobs over the S3/GCS events table: a blob job (one row per object version; bytes inline up to a threshold, else an unmanaged out-of-line reference to the source object in place, nothing copied into the table) and an embedding job (text chunks, images, video frames and speech transcripts, with one VECTOR column per configured model). It also proposes two generic, opt-in Hudi Streamer additions (a source contract and an instant-aware transformer), lists the edits to existing Hudi code, the limits, the native/third-party dependencies (FFmpeg optional and never bundled) and the PR plan. - `rfc/README.md`: RFC-118 row, `UNDER REVIEW`. Proposed approvers: @yihua, @vinothchandar. ### Impact None: documentation only. The changes the RFC proposes are listed in its "Edits to existing Hudi code" and "Rollout/Adoption Plan" sections. ### Risk Level none ### Documentation Update none: this PR is the design document. Config documentation comes with each code PR. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
