Davis-Zhang-Onehouse opened a new issue, #20683: URL: https://github.com/apache/hudi/issues/20683
### Feature Description **What the feature achieves:** Tracking issue for RFC-118. Adds `UnstructuredDataSource` to Hudi Streamer with two modes over the S3/GCS events table the stock events pipeline already builds: - **blob**: one row per object version (modification time, size). Bytes are stored inline up to `hoodie.blob.table.inline.max.bytes`; a larger object is an unmanaged out-of-line reference (`managed=false`) to the source object in place. Nothing is copied into the table. The trade-offs of referencing in place are listed in RFC-118 ยง2. - **embedding**: prepares files into text chunks, images, video frames and speech transcripts (`prep_only`), and optionally computes embeddings with configured models into one VECTOR column per model (`prep_and_compute`). The only files it writes beside data files are extracted video frames, inside the round's instant, so a failed round rolls them back. Builds on RFC-100 (BLOB type, read_blob) and runs alongside the existing unstructured ingestion (#19278); produces tables usable with RFC-102/RFC-109 vector search. **Why this feature is needed:** To keep blob and embedding tables up to date incrementally from object-storage events, once per file version, with bounded decoding of PDF, image and video inputs and bad files quarantined in the error table instead of failing the pipeline. ### User Experience **How users will use this feature:** - Configuration changes needed: `--source-class org.apache.hudi.utilities.sources.UnstructuredDataSource`, `hoodie.unstructured.source.mode` (`blob` | `embedding`), `hoodie.unstructured.source.cloud` (`s3` | `gcs`); for embedding, `hoodie.unstructured.embedding.mode` and the model keys. The source contract fills in the required transformer, record key, Parquet injector and marker settings, and refuses conflicting ones. - API changes: two generic Hudi Streamer additions, inert for existing sources and transformers: a source contract and an instant-aware transformer. - Usage examples: in RFC-118. The implementation lands as a chain of PRs, each compiling and tested on its own; each code PR opens its own sub-issue of this issue. PRs after the first segment open as drafts and become ready as their predecessor merges. Sub-issues 4, 10, 27, 28, 29, 31, 32, 34 and 36 of the plan chiefly carry a blob copy path (large objects copied under `.hoodie/.blob`) that follow-up 37 replaces with the in-place reference RFC-118 describes; the end state is the chain plus 37. Sub-issue 38 is a generic memory fix, independent of the chain. ### Hudi RFC Requirements **RFC PR link:** RFC-118 (the RFC PR references this issue) **Why RFC is/isn't needed:** - Does this change public interfaces/APIs? Yes (a new Streamer source and two Streamer extension points) - Does this change storage format? No (BLOB and VECTOR columns as RFC-100/RFC-102 define them; table-config properties of the new tables) - Justification: a new Hudi Streamer source with generic streamer additions needs an RFC. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
