Davis-Zhang-Onehouse opened a new pull request, #20684:
URL: https://github.com/apache/hudi/pull/20684

   ### Describe the issue this Pull Request addresses
   
   Adds RFC-118, the design for ingesting unstructured files (PDF, images, 
video) announced by S3/GCS event notifications into Hudi tables with Hudi 
Streamer, and for keeping per-input embedding tables over them up to date. 
Feature issue: #20683.
   
   ### Summary and Changelog
   
   - `rfc/rfc-118/rfc-118.md`: the RFC. It proposes one new source, 
`UnstructuredDataSource`, with two jobs over the S3/GCS events table: a blob 
job (one row per object version; bytes inline up to a threshold, else an 
unmanaged out-of-line reference to the source object in place, nothing copied 
into the table) and an embedding job (text chunks, images, video frames and 
speech transcripts, with one VECTOR column per configured model). It also 
proposes two generic, opt-in Hudi Streamer additions (a source contract and an 
instant-aware transformer), lists the edits to existing Hudi code, the limits, 
the native/third-party dependencies (FFmpeg optional and never bundled) and the 
PR plan.
   - `rfc/README.md`: RFC-118 row, `UNDER REVIEW`.
   
   Proposed approvers: @yihua, @vinothchandar.
   
   ### Impact
   
   None: documentation only. The changes the RFC proposes are listed in its 
"Edits to existing Hudi code" and "Rollout/Adoption Plan" sections.
   
   ### Risk Level
   
   none
   
   ### Documentation Update
   
   none: this PR is the design document. Config documentation comes with each 
code PR.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to