Davis-Zhang-Onehouse opened a new pull request, #20707: URL: https://github.com/apache/hudi/pull/20707
### Describe the issue this Pull Request addresses issue: #20706 Part of #20683, RFC-118 (#20684). Stacked on #20700: review only the last commit. Until the PRs below it merge, this PR also shows their commits. The `prep_and_compute` embedding round: one Hudi Streamer round prepares the new files into chunks and computes their embeddings with the configured models, writing one VECTOR column per model, with the PDF path working end to end. It covers in-step and task retries of the compute step without re-running prep, provider outages, error-table rows for failed files, one commit per round, and blob and embedding streams running concurrently. ### Summary and Changelog - The `prep_and_compute` mode of `EmbeddingTableTransformer`: prep and compute in one round, one commit, one VECTOR column per configured model. - Compute-step behaviour: in-step retries, task retries that do not re-run prep, shuffle balance, provider outages surfacing as round failures, and failed files routed to the error table. - `UnstructuredDataSource` end-to-end coverage: S3 and GCS events, target schema, ordering field, customer transformers, embedding from S3 events without a blob table, overwrite convergence, and blob and embedding streams side by side. - 27 test classes (23 new). Test-only adaptations to master: cases for a delete option master does not have are removed, compression is read from master's writer configuration, and the two-stream concurrency test turns off the column-stats index of its source-events fixture. With that index on, two streams in one Spark session can deadlock inside Spark's cache manager (`CacheManager.uncacheQuery`), reached through data skipping. That is outside this feature; production code is unchanged and the fixture still checks both streams' data, checkpoints and provider failures. All files are in hudi-utilities and were added earlier in this chain or here. No code was copied from other projects. Known limitation, closed by the next PR (blob rollback): a failed blob round can still leave unreferenced copies under `.hoodie/.blob`. ### Impact `UnstructuredDataSource` in embedding mode can now compute embeddings, not only prepare inputs. Existing sources are unaffected. ### Risk Level medium: first end-to-end compute round of the new source. Verified on JDK 11 / Spark 3.5 / Scala 2.12 with 155 tests (none skipped), module compilation, checkstyle and RAT. ### Documentation Update The embedding mode settings carry their config documentation. RFC-118 describes the round; a website page follows once the chain lands. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
