Davis-Zhang-Onehouse opened a new pull request, #20733:
URL: https://github.com/apache/hudi/pull/20733

   ### Describe the issue this Pull Request addresses
   
   issue: #20732
   Part of #20683, RFC-118 (#20684).
   Stacked on #20731: review only the last commit. Until the PRs below it 
merge, this PR also shows their commits.
   
   Embedding clients for Amazon Bedrock (Titan and Cohere text models, signed 
with the AWS SDK default credential chain), Google Vertex AI (text and 
multimodal, with token refresh and its retry loop) and Jina, each with its 
compute writer. Also closes correctness and resource-limit gaps found in review 
of the earlier PRs of this chain.
   
   ### Summary and Changelog
   
   - Embedding clients with compute writers: Amazon Bedrock (Titan and Cohere 
text models), Google Vertex AI (text and multimodal, token refresh, retry loop) 
and Jina.
   - Adds `software.amazon.awssdk:bedrockruntime` to hudi-utilities. Like the 
other AWS SDK clients there (sqs, kinesis, sts), it is not shaded into the 
utilities bundles: a job that uses the Bedrock provider adds the AWS SDK jars 
(bedrockruntime, its SDK core dependencies and apache-client) at launch. 
Requests are signed with the AWS SDK default credential chain; there is no 
credential key in the embedding config.
   
   This PR also closes correctness and resource-limit gaps in the earlier PRs 
of this chain, each with a regression test, a passing control and a 
revert-to-red run:
   - Compute rounds keep customer columns whose names match the internal 
outcome column, including case variants.
   - A denied source read fails the round and keeps its checkpoint, instead of 
being treated as a missing file.
   - PDF extraction charges repeated font materializations, scans the decoder 
bytes the parser actually consumes, and records a truncated empty page as a 
durable loss. The PDF record implementation version advances to v3.
   - Older embedding tables gain the nullable image BLOB column before prep or 
compute.
   - Image preparation admits source-width PNG/BMP/GIF row buffers before 
subsampling, and reads PNG row layout without copying unrelated metadata.
   - 12 provider test classes plus the regression tests; recorded responses 
under `unstructured/vertex/`; no live endpoint is called.
   
   No code was copied from other projects.
   
   ### Impact
   
   The embedding stream can call Bedrock, Vertex AI and Jina. hudi-utilities 
gains the bedrockruntime dependency (not bundled). Fixes listed above change 
behaviour only inside the new source.
   
   ### Risk Level
   
   medium: new provider dependency and fixes across the earlier stages. 
Verified on JDK 11 / Spark 3.5 / Scala 2.12 with 339 tests (none skipped), 
module compilation, checkstyle and RAT. The integrated chain up to this PR 
passed 1987 tests with 0 failures or errors and 10 accepted, gated skips; all 
24 selected memory cells passed; both bundles were checked.
   
   ### Documentation Update
   
   New provider settings carry their config documentation; a website page 
follows once the chain lands.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(ingest): RFC-118: ... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to