Davis-Zhang-Onehouse opened a new pull request, #20694:
URL: https://github.com/apache/hudi/pull/20694

   ### Describe the issue this Pull Request addresses
   
   issue: #20693
   Part of #20683, RFC-118 (#20684).
   Stacked on #20692: review only the last commit. Until the PRs below it 
merge, this PR also shows their commits.
   
   Adds the embedding table contract that the later RFC-118 preprocessing and 
compute stages use, and the dependencies those stages need.
   
   ### Summary and Changelog
   
   - Embedding table core: `EmbeddingTableConfig`, `EmbeddingTableSchema`, 
`EmbeddingModelSpec`, `EmbeddingTableChecks`, `EmbeddingTableVersionException` 
(key schema, the `prep_only` table shape, column names and schema checks).
   - Embedding client interface (`EmbeddingClient`, `EmbeddingInput`, 
`EmbeddingResult`, `EmbeddingClientNames`) and `DummyEmbeddingClient` for tests.
   - `BlobColumns` (shared BLOB column names) and `TransientRetry`.
   - New dependencies, pinned in the root `pom.xml`: Apache PDFBox 3.0.8 (with 
FontBox) and jtokkit 1.1.0 (MIT). Both are added to `hudi-utilities` and to the 
utilities and utilities-slim bundles, relocated under `org.apache.hudi`.
   - `LICENSE` / `NOTICE`: blocks for PDFBox/FontBox and their notices 
(including the Unicode license v3 text they carry), jtokkit (MIT) and the 
openai/tiktoken vocabulary files bundled in jtokkit (MIT). The bundles keep 
Hudi's existing LICENSE convention, as other shaded dependencies do.
   - 7 test files.
   
   No source code was copied from other projects; the new attributions are for 
bundled binaries and data.
   
   ### Impact
   
   Adds classes used by later stages; no behaviour of existing features 
changes. The utilities bundles grow by the two relocated dependencies.
   
   ### Risk Level
   
   medium: new dependencies and bundle relocations. Verified on JDK 11 / Spark 
3.5 / Scala 2.12 with 37 tests, module compilation, checkstyle and RAT. Both 
utilities bundles were rebuilt; in each, the relocated tokenizer loads its 
bundled vocabularies and round-trips text.
   
   ### Documentation Update
   
   New configuration fields carry their documentation. RFC-118 describes the 
embedding table; a website page follows once the chain lands.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(ingest): RFC-118: ... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to