Davis-Zhang-Onehouse opened a new pull request, #20731:
URL: https://github.com/apache/hudi/pull/20731

   ### Describe the issue this Pull Request addresses
   
   issue: #20730
   Part of #20683, RFC-118 (#20684).
   Stacked on #20729: review only the last commit. Until the PRs below it 
merge, this PR also shows their commits.
   
   An embedding provider base (request rate limiting, rejection policy, 
availability checks and per-call statistics) and the OpenAI-compatible 
embedding client (OpenAI, Azure OpenAI v1 and Hugging Face TEI style endpoints) 
with its compute writer.
   
   ### Summary and Changelog
   
   - Provider base: request rate limiting, a rejection policy for provider 
errors, text-provider availability checks and per-call statistics.
   - `OpenAiCompatibleEmbeddingClient` for OpenAI, Azure OpenAI v1 and Hugging 
Face TEI style endpoints, with its compute writer and rate-limit handling.
   - 10 test classes (9 new). Tests use recorded response shapes under 
`unstructured/openai/`; no live endpoint is called.
   
   No code was copied from other projects.
   
   ### Impact
   
   The embedding stream can call OpenAI-compatible embedding endpoints. No 
change outside the new source.
   
   ### Risk Level
   
   low: new provider code of the new source. Verified on JDK 11 / Spark 3.5 / 
Scala 2.12 with 136 tests (none skipped), module compilation, checkstyle and 
RAT.
   
   ### Documentation Update
   
   New provider settings carry their config documentation; a website page 
follows once the chain lands.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(ingest): RFC-118: ... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to